Cutting-plane Method and Its Oracles

---

# 🥥 Cutting-plane Method and Its Oracles

Wai-Shing Luk

http://platon.io/Iph8

2017-10-26 📅

---

> When you have eliminated the impossible, whatever remains, however
> improbable, must be the truth.

*Sir Arthur Conan Doyle, stated by Sherlock Holmes*

---

# 👋 Introduction

---

### 🏛️ Some History of Ellipsoid Method

- Introduced by Shor and Yudin and Nemirovskii in 1976

- Used to show that linear programming (LP) is polynomial-time
    solvable (Kachiyan 1979), settled the long-standing problem of
    determining the theoretically complexity of LP.

- In practice, however, the simplex method runs much faster than the
    method, although its worst-case complexity is exponential.

---

### 👀 Common Perspective of Ellipsoid Method

- It is commonly believed that it is inefficient in practice for
    large-scale problems.

- The convergent rate is slow, even with the use of deep cuts.

- Cannot exploit sparsity.

- Since then, it was supplanted by interior-point methods.

- Only treated as a theoretical tool for proving the polynomial-time
    solvability of combinatorial optimization problems.

---

### But...🤔

- The ellipsoid method works very differently compared with the
    interior point method.

- Only require a cutting-plane oracle. Can play nicely with
    other techniques.

- The oracle can exploit sparsity.

---

### Consider Ellipsoid Method When...🤔

- The number of optimization variables is moderate, e.g. ECO flow,
    analog circuit sizing, parametric problems

- The number of constraints is large, or even infinite

- Oracle can be implemented efficiently.

---

# 🥥 Cutting-plane Method Revisited

---

### Basic Idea 🥚

- Let $\mathcal{\color{steelblue}K} \subseteq \mathbb{R}^n$ be a convex set.
- Consider the feasibility problem:
  - Find a point ${\color{darkblack}x}^* \in \mathbb{R}^n$ in $\mathcal{\color{steelblue}K}$,
  - or determine that $\mathcal{\color{steelblue}K}$ is empty (i.e., no
        feasible sol'n)

]
.pull-right[

![](lec06.files/region.svg)

]

---

### 🥥 Cutting-plane Oracle

- When cutting-plane oracle $\Omega$ is *queried* at ${\color{darkblack}x}_0$, it either
  - asserts that ${\color{darkblack}x_0} \in \mathcal{\color{steelblue}K}$, or
  - returns a separating hyperplane between ${\color{darkblack}x}_0$ and $\mathcal{\color{steelblue}K}$:
        $${\color{darkcyan} g^\mathsf{T} } ({\color{darkblack}x} - {\color{darkblack}x_0}) + {\color{green} h} \leq 0, {\color{green} h} \geq 0, {\color{darkcyan} g} \neq 0, \;
          \forall {\color{darkblack}x} \in \mathcal{\color{steelblue}K}$$

]
.pull-right30[

![](lec06.files/cut.svg)

]

---

### 🥥 Cutting-plane oracle (cont'd)

- $({\color{darkcyan} g},{\color{green} h})$ called a *cutting-plane*, or cut, since it eliminates the
    halfspace $\{{\color{darkblack}x} \mid {\color{darkcyan} g^\mathsf{T} } ({\color{darkblack}x} - {\color{darkblack}x_0}) + {\color{green} h} > 0\}$ from our search.

- If ${\color{green} h}=0$ (${\color{darkblack}x}_0$ is on boundary of halfspace that is cut),
    cutting-plane is called *neutral cut*.

- If ${\color{green} h}>0$ (${\color{darkblack}x}_0$ lies in interior of halfspace that is cut),
    cutting-plane is called *deep cut*.

---

### Subgradient 📉

- $\mathcal{\color{steelblue}K}$ is usually given by a set of inequalities
    ${\color{purple}f}_j({\color{darkblack}x}) \le 0$ or ${\color{purple}f}_j({\color{darkblack}x}) < 0$ for $j = 1 \cdots m$, where ${\color{purple}f}_j({\color{darkblack}x})$
    is a convex function.
- A vector ${\color{darkcyan} g} \equiv \partial {\color{purple}f}({\color{darkblack}x_0})$ is called a subgradient of a
    convex function ${\color{purple}f}$ at ${\color{darkblack}x}_0$ if
    ${\color{purple}f}({\color{darkblack}z}) \geq {\color{purple}f}({\color{darkblack}x_0}) + {\color{darkcyan} g^\mathsf{T} } ({\color{darkblack}z} - {\color{darkblack}x_0})$.
- Hence, the cut $({\color{darkcyan} g},{\color{green} h})$ is given by $(\partial {\color{purple}f}({\color{darkblack}x_0}), {\color{purple}f}({\color{darkblack}x_0}))$

Remarks:

- If ${\color{purple}f}({\color{darkblack}x})$ is differentiable, we can simply take
    $\partial {\color{purple}f}({\color{darkblack}x_0}) = \nabla {\color{purple}f}({\color{darkblack}x_0})$

---

### Key components of Cutting-plane method 🗝️

- Cutting plane oracle $\Omega$
- A search space ${\color{red}\mathcal{S} }$ initially big enough to cover $\mathcal{\color{steelblue}K}$, e.g.
  - Polyhedron ${\color{red}\mathcal{P} }$ = $\{{\color{darkblack}z} \mid {\color{coral}C} {\color{darkblack}z} \preceq {\color{coral}d} \}$
  - Interval ${\color{red}\mathcal{I} }$ = $[{\color{coral}l}, {\color{coral}u}]$ (for one-dimensional problem)
  - Ellipsoid ${\color{red}\mathcal{E} }$ = $\{{\color{darkblack}z} \mid ({\color{darkblack}z}-{\color{orange}x_c}){\color{red}P}^{-1}({\color{darkblack}z}-{\color{orange}x_c}) \leq 1 \}$

---

### ⚙️ Generic Cutting-plane method

- **Given** initial ${\color{red}\mathcal{S} }$ known to contain $\mathcal{\color{steelblue}K}$.
- **Repeat**
  1. Choose a point ${\color{darkblack}x}_0$ in ${\color{red}\mathcal{S} }$
  2. Query the cutting-plane oracle at ${\color{darkblack}x}_0$
  3. **If** ${\color{darkblack}x_0} \in \mathcal{\color{steelblue}K}$, quit
4. **Else**, update ${\color{red}\mathcal{S} }$ to a smaller set that covers:
        $${\color{violet}\mathcal{S}^+} = {\color{red}\mathcal{S} } \cap \{{\color{darkblack}z} \mid {\color{darkcyan} g^\mathsf{T} } ({\color{darkblack}z} - {\color{darkblack}x_0}) + {\color{green} h} \leq 0\}$$
5. **If** ${\color{violet}\mathcal{S}^+} = \emptyset$ or it is small enough, quit.

---

### Cutting-plane Method Flow 🔄

.mermaid[
<pre>
flowchart TD
    style A fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
    style B fill:#e3f2fd,stroke:#1565c0,stroke-width:3px
    style C fill:#fff3e0,stroke:#e65100,stroke-width:3px
    style D fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
    style E fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
    style F fill:#ffcdd2,stroke:#c62828,stroke-width:3px
    style G fill:#e3f2fd,stroke:#1565c0,stroke-width:3px
    style H fill:#fff3e0,stroke:#e65100,stroke-width:3px
    style I fill:#ffcdd2,stroke:#c62828,stroke-width:3px

A(("Initial polyhedron S")) --> B[Choose point x₀ in S]
    B --> C[Query cutting-plane oracle at x₀]
    C --> D{x₀ ∈ K?}
    D -->|Yes| E([Return x₀ ✓])
    D -->|No| F[Get cutting plane g,h]
    F --> G[Update: shrink polyhedron]
    G --> H{S empty or small?}
    H -->|Yes| I([No solution])
    H -->|No| B
</pre>
]

---

### 🐍 Python code

```python
def cutting_plane_fea(assess, S, t, max_it, tol):
    flag = 0
    for iter in range(max_it):
        g, h, t1 = assess(S.xc, t)
        if t != t1: # feasible sol'n obtained
            flag = 1
            break
        status, tau = S.update(g, h)
        if status == 1: break
        if tau < tol:
            status = 2
            break
    return S.xc, iter, flag, status
```

---

### 🥚 Convex Optimization Problem (I)

$$\begin{array}{ll}
    \text{minimize}     & {\color{purple}f}_0({\color{darkblack}x}), \\
    \text{subject to}   & {\color{darkblack}x} \in \mathcal{\color{steelblue}K}
\end{array}$$

- The optimization problem is treated as a feasibility problem with an
    additional constraint ${\color{purple}f}_0({\color{darkblack}x}) < {\color{coral}t}$

- ${\color{purple}f}_0({\color{darkblack}x})$ could be a convex function or a quasiconvex function.

- ${\color{coral}t}$ is the best-so-far value of ${\color{purple}f}_0({\color{darkblack}x})$.

---

### 🥚 Convex Optimization Problem (II)

- Problem formulation:
    $$\begin{array}{ll}
              \text{find}         & {\color{darkblack}x}, \\
              \text{subject to}   & {\color{purple}\Phi}_{\color{coral}t}({\color{darkblack}x}) < 0 \\
                                  & {\color{darkblack}x} \in \mathcal{\color{steelblue}K}
    \end{array}$$ where ${\color{purple}\Phi}_{\color{coral}t}({\color{darkblack}x}) < 0$ is the ${\color{coral}t}$-sublevel set of ${\color{purple}f}_0({\color{darkblack}x})$.
- 👉 Note: $\mathcal{\color{steelblue}K}_{\color{coral}t} \subseteq \mathcal{\color{steelblue}K}_{\color{coral}u}$ if and only if
    ${\color{coral}t} \leq {\color{coral}u}$ (monotonicity)

---

### 🥚 Convex Optimization Problem (III)

- One easy way to solve the optimization problem is to apply the binary
    search on ${\color{coral}t}$.
- Another possible way is, to update the best-so-far ${\color{coral}t}$ whenever a
    feasible sol'n ${\color{darkblack}x}_0$ is found such that ${\color{purple}\Phi}_{\color{coral}t}({\color{darkblack}x_0}) = 0$.
- We assume that the oracle takes the responsbility for that.

---

### ⚙️ Generic Cutting-plane method (Optim)

- **Given** initial ${\color{red}\mathcal{S} }$ known to contain $\mathcal{\color{steelblue}K}_{\color{coral}t}$.
- **Repeat**
  1. Choose a point ${\color{darkblack}x}_0$ in ${\color{red}\mathcal{S} }$
  2. Query the cutting-plane oracle at ${\color{darkblack}x}_0$
  3. **If** ${\color{darkblack}x_0} \in \mathcal{\color{steelblue}K}_{\color{coral}t}$, update ${\color{coral}t}$ such that
        ${\color{purple}\Phi}_{\color{coral}t}({\color{darkblack}x_0}) = 0$.
  4. Update ${\color{red}\mathcal{S} }$ to a smaller set that covers:
        $${\color{violet}\mathcal{S}^+} = {\color{red}\mathcal{S} } \cap \{{\color{darkblack}z} \mid {\color{darkcyan} g^\mathsf{T} } ({\color{darkblack}z} - {\color{darkblack}x_0}) + {\color{green} h} \leq 0\} $$
  5. **If** ${\color{violet}\mathcal{S}^+} = \emptyset$ or it is small enough, quit.

---

### 🐍 Python code (II)

```python
import numpy as np

def cutting_plane_dc(assess, S, t, max_it, tol):
    flag = 0 # no sol'n
    for iter in range(max_it):
        g, h, t1 = assess(S.xc, t)
        if t != t1: # best t obtained
            flag, t = 1, t1
            x_best = np.array(S.xc)
        status, tau = S.update(g, h)
        if status == 1: break
        if tau < tol:
            status = 2
            break
    return x_best, t, iter, flag, status
```

---

### 📚 Example: Profit Maximization Problem

$$\begin{array}{ll}
   \text{maximize} & {\color{coral}p}({\color{coral}A} {\color{darkblack}x}_1^{\color{coral}\alpha} {\color{darkblack}x}_2^{\color{coral}\beta}) - {\color{coral}v_1} {\color{darkblack}x}_1 - {\color{coral}v_2} {\color{darkblack}x}_2 \\
   \text{subject to}& {\color{darkblack}x}_1 \le {\color{coral}k}.
\end{array}$$

- ${\color{coral}p}({\color{coral}A} {\color{darkblack}x}_1^{\color{coral}\alpha} {\color{darkblack}x}_2^{\color{coral}\beta})$ : Cobb-Douglas production function
- ${\color{coral}p}$: the market price per unit
- ${\color{coral}A}$: the scale of production
- ${\color{coral}\alpha}, {\color{coral}\beta}$: the output elasticities
- ${\color{darkblack}x}$: input quantity
- ${\color{coral}v}$: output price
- ${\color{coral}k}$: a given constant that restricts the quantity of ${\color{darkblack}x}_1$

---

### 📚 Example: Profit maximization (cont'd)

- The formulation is not in the convex form.
- Rewrite the problem in the following form:
    $$\begin{array}{ll}
    \text{maximize} & {\color{coral}t} \\
    \text{subject to} & {\color{coral}t}  + {\color{coral}v_1} {\color{darkblack}x}_1  + {\color{coral}v_2} {\color{darkblack}x}_2 < {\color{coral}p} {\color{coral}A} {\color{darkblack}x}_1^{\color{coral}\alpha} {\color{darkblack}x}_2^{\color{coral}\beta}\\
                  & {\color{darkblack}x}_1 \le {\color{coral}k}.
    \end{array}$$

---

### 💰 Profit maximization in Convex Form

- Change variables to:

- ${\color{darkblack}y_1} = \log {\color{darkblack}x}_1$, ${\color{darkblack}y_2} = \log {\color{darkblack}x}_2$.

and take logarithm of cost and constraints.

- We have the problem in a convex form:
    $$\begin{array}{ll}
    \text{max}  & {\color{coral}t} \\
    \text{s.t.} & \log({\color{coral}t} + {\color{coral}v_1} e^{{\color{darkblack}y_1}} + {\color{coral}v_2} e^{{\color{darkblack}y_2}}) - ({\color{coral}\alpha} {\color{darkblack}y_1} + {\color{coral}\beta} {\color{darkblack}y_2}) < \log({\color{coral}p}{\color{coral}A})  \\
                  & {\color{darkblack}y_1} \le \log {\color{coral}k} ,
    \end{array}$$

---

### 🐍 Python code (Profit oracle)

```python
import numpy as np
class: profit_oracle:
    def __init__(self, p, A, alpha, beta, v1, v2, k):...
    def assess_optim(self, y, t):
        fj = y[0] - self.log_k; # constraint
        if fj > 0.0:
            g = np.array([1.0, 0.0])
            return g, fj, t
        log_Cobb = self.log_pA + np.dot(self.a, y)
        x = np.exp(y)
        vx = np.dot(self.v, x)
        te = t + vx
        fj = np.log(te) - log_Cobb
        if fj < 0.0:
            t = np.exp(log_Cobb) - vx
            te = t + vx
            fj = 0.0
        g = (self.v * x)/te - self.a
        return g, fj, t
```

---

### 🐍 Python code (Main program)

```python
import numpy as np
from profit_oracle import *
from cutting_plane import *
from ell import *

p, A, k = 20.0, 40.0, 30.5
alpha, beta = 0.1, 0.4
v1, v2 = 10.0, 35.0
y0 = np.array([0.0, 0.0])  # initial x0
E = ell(200, y0)
P = profit_oracle(p, A, alpha, beta, v1, v2, k)
yb1, fb, iter, flag, status = cutting_plane_dc(P, E, 0.0, 200, 1e-8)
print(fb, iter, flag, status)
```

---

### Area of Applications 🌍

- 🛡️ Robust convex optimization
  - oracle technique: affine arithmetic
- Parametric network potential problem
  - oracle technique: negative cycle detection
- Semidefinite programming
  - oracle technique: Cholesky factorization

---

# 🛡️ Robust Convex Optimization

---

### 🛡️ Robust Optimization Formulation

- Consider:
    $$\begin{array}{ll}
      \text{minimize}   & \sup_{{\color{coral}q} \in {\mathbb{Q} } } {\color{purple}f_0}({\color{darkblack}x},{\color{coral}q}) \\
      \text{subject to} & {\color{purple}f_j}({\color{darkblack}x},{\color{coral}q}) \leq 0 \qquad \\
      & \forall {\color{coral}q} \in {\mathbb{Q} } \; \text{and} \; j = 1,2,\cdots,m,
    \end{array}$$ where ${\color{coral}q}$ represents a set of varying parameters.

- The problem can be reformulated as:
    $$\begin{array}{ll}
      \text{minimize}   & {\color{coral}t} \\
      \text{subject to} & {\color{purple}f_0}({\color{darkblack}x},{\color{coral}q}) < {\color{coral}t}  \\
      & {\color{purple}f_j}({\color{darkblack}x},{\color{coral}q}) \leq 0 \\
      & \forall {\color{coral}q} \in {\mathbb{Q} } \; \text{and} \; j = 1,2,\cdots,m,
    \end{array}$$

---

### 🔮 Oracle in Robust Optimization Formulation

- The oracle only needs to determine:
    - If ${\color{purple}f_j}({\color{darkblack}x}_0, {\color{coral}q}) > 0$ for some $j$ and ${\color{coral}q} = {\color{coral}q}_0$, then
        - the cut $({\color{darkcyan} g},{\color{green} h})$ = $(\partial {\color{purple}f_j}({\color{darkblack}x}_0, {\color{coral}q}_0), {\color{purple}f_j}({\color{darkblack}x}_0, {\color{coral}q}_0))$
    - If ${\color{purple}f_0}({\color{darkblack}x}_0, {\color{coral}q}) \geq {\color{coral}t}$ for some ${\color{coral}q} = {\color{coral}q}_0$, then
        - the cut $({\color{darkcyan} g},{\color{green} h})$ = $(\partial {\color{purple}f_0}({\color{darkblack}x}_0, {\color{coral}q}_0), {\color{purple}f_0}({\color{darkblack}x}_0, {\color{coral}q}_0) - {\color{coral}t})$
    - Otherwise, ${\color{darkblack}x}_0$ is feasible, then
        - Let ${\color{coral}q}_{\max} = \argmax_{{\color{coral}q} \in {\mathbb{Q} } } {\color{purple}f_0}({\color{darkblack}x}_0, {\color{coral}q})$.
        - ${\color{coral}t} := {\color{purple}f_0}({\color{darkblack}x}_0, {\color{coral}q}_{\max})$.
        - The cut $({\color{darkcyan} g},{\color{green} h})$ = $(\partial {\color{purple}f_0}({\color{darkblack}x}_0, {\color{coral}q}_{\max}), 0)$

---

### 📚 Example: Profit Maximization Problem (convex)

$$\begin{array}{ll}
\text{max}  & {\color{coral}t} \\
\text{s.t.} & \log({\color{coral}t} + {\color{coral}\hat{v}}_1 e^{{\color{darkblack}y}_1} + {\color{coral}\hat{v}}_2 e^{{\color{darkblack}y}_2}) - ({\color{coral}\hat{\alpha}} {\color{darkblack}y}_1 + {\color{coral}\hat{\beta}} {\color{darkblack}y}_2) \le \log({\color{coral}\hat{p}}\,{\color{coral}A})  \\
                  & {\color{darkblack}y}_1 \le \log {\color{coral}\hat{k}} ,
\end{array}$$

- Now assume that:
  - ${\color{coral}\hat{\alpha}}$ and ${\color{coral}\hat{\beta}}$ vary ${\color{coral}\bar{\alpha}} \pm {\color{coral}e}_1$ and
        ${\color{coral}\bar{\beta}} \pm {\color{coral}e}_2$ respectively.
  - ${\color{coral}\hat{p}}$, ${\color{coral}\hat{k}}$, ${\color{coral}\hat{v}}_1$, and ${\color{coral}\hat{v}}_2$ all vary
        $\pm {\color{coral}e}_3$.

---

### 📚 Example: Profit Maximization Problem (oracle)

By detail analysis, the worst case happens when:

- ${\color{coral}p} = {\color{coral}\bar{p}} + {\color{coral}e}_3$, ${\color{coral}k} = {\color{coral}\bar{k}} + {\color{coral}e}_3$
- ${\color{coral}v}_1 = {\color{coral}\bar{v}}_1 - {\color{coral}e}_3$, ${\color{coral}v}_2 = {\color{coral}\bar{v}}_2 - {\color{coral}e}_3$,
- if ${\color{darkblack}y}_1 > 0$, ${\color{coral}\alpha} = {\color{coral}\bar{\alpha}} - {\color{coral}e}_1$, else
    ${\color{coral}\alpha} = {\color{coral}\bar{\alpha}} + {\color{coral}e}_1$
- if ${\color{darkblack}y}_2 > 0$, ${\color{coral}\beta} = {\color{coral}\bar{\beta}} - {\color{coral}e}_2$, else
    ${\color{coral}\beta} = {\color{coral}\bar{\beta}} + {\color{coral}e}_2$

> **Remark**: for more complicated problems, affine arithmetic could be
> used.

---

### 🐍 Python code (profit_rb_oracle)

```python
class: profit_rb_oracle:
    def __init__(self, p, A, alpha, beta, v1, v2, k, ui, e1, e2, e3):...
    def assess_optim(self, y, t):
        fj = y[0] - self.log_k; # constraint
        if fj > 0.0:
            g = np.array([1.0, 0.0])
            return g, fj, t
        a_rb = np.array(self.a)
        a_rb[0] += self.uie1 * (+1.0 if y[0] <= 0.0 else -1.0)
        a_rb[1] += self.uie2 * (+1.0 if y[1] <= 0.0 else -1.0)
        log_Cobb = self.log_pA + np.dot(a_rb, y)
        x = np.exp(y)
        vx = np.dot(self.v, x)
        te = t + vx
        fj = np.log(te) - log_Cobb
        if fj < 0.0:
            t = np.exp(log_Cobb) - vx
            te = t + vx
            fj = 0.0
        g = (self.v * x)/te - a_rb
        return g, fj, t
```

---

# Parametric Network Potential Problem

---

### 🌐 Parametric Network Potential Problem

Given a network represented by a graph $G = ({\color{salmon}V}, {\color{lime}E})$. Consider

$$\begin{array}{ll}
    \text{minimize} & {\color{purple}f_0}({\color{darkblack}x}) \\
    \text{subject to} & {\color{darkblack}u_i} - {\color{darkblack}u_j} \le {\color{purple}h}_{ij}({\color{darkblack}x}), \; \forall (i, j) \in {\color{lime}E} ,\\
    \text{variables} &{\color{darkblack}x}, {\color{darkblack}u} ,
   \end{array}$$

- ${\color{purple}h}_{ij}({\color{darkblack}x})$ is the weight function of edge $(i,j)$,

- Assume: network is large but the number of parameters is small.

---

### 🌐 Network Potential Problem (cont'd)

Given ${\color{darkblack}x}_0$, the problem has a feasible solution if and only if $G$
contains no negative cycle. Let ${\color{lime}\mathcal{C} }$ be a set of all cycles of
$G$.

$$\begin{array}{ll}
    \text{minimize} & {\color{purple}f_0}({\color{darkblack}x}) \\
    \text{subject to} & {\color{purple}W}_k({\color{darkblack}x}) \ge 0, \forall {\color{lime}C}_k \in {\color{lime}C} ,\\
       \text{variables} & {\color{darkblack}x}
   \end{array}$$

- ${\color{lime}C}_k$ is a cycle of $G$

- ${\color{purple}W}_k({\color{darkblack}x}) = \sum_{ (i,j)\in {\color{lime}C}_k} {\color{purple}h}_{ij}({\color{darkblack}x})$.

---

### 🔮 Oracle in Network Potential Problem

- The oracle only needs to determine:
    - If there exists a negative cycle ${\color{lime}C}_k$ under ${\color{darkblack}x}_0$, then
        - the cut $({\color{darkcyan} g},{\color{green} h})$ = $(-\partial {\color{purple}W}_k({\color{darkblack}x}_0), -{\color{purple}W}_k({\color{darkblack}x}_0))$
    - If ${\color{purple}f_0}({\color{darkblack}x}_0) \geq {\color{coral}t}$, then
        - the cut $({\color{darkcyan} g},{\color{green} h})$ = $(\partial {\color{purple}f_0}({\color{darkblack}x}_0), {\color{purple}f_0}({\color{darkblack}x}_0) - {\color{coral}t})$
    - Otherwise, ${\color{darkblack}x}_0$ is feasible, then
        - ${\color{coral}t} := {\color{purple}f_0}({\color{darkblack}x}_0)$.
        - The cut $({\color{darkcyan} g},{\color{green} h})$ = $(\partial {\color{purple}f_0}({\color{darkblack}x}_0), 0)$

---

### 📚 Example: Optimal Matrix Scaling

- Given a sparse matrix ${\color{coral}A} = [{\color{coral}a}_{ij}] \in \mathbb{R}^{N\times N}$.

- Find another matrix ${\color{coral}B} = {\color{darkblack}U} {\color{coral}A} {\color{darkblack}U}^{-1}$ where ${\color{darkblack}U}$ is a nonnegative
    diagonal matrix, such that the ratio of any two elements of $B$ in
    absolute value is as close to 1 as possible.

- Let ${\color{darkblack}U} = \mathrm{diag}([{\color{darkblack}u}_1, {\color{darkblack}u}_2, \ldots, {\color{darkblack}u}_N])$. Under the
    min-max-ratio criterion, the problem can be formulated as:

$$\begin{array}{ll}
  \text{minimize}   &   {\color{darkblack}\pi}/{\color{darkblack}\psi}  \\
  \text{subject to} &   {\color{darkblack}\psi} \leq {\color{darkblack}u}_i |{\color{coral}a}_{ij}| {\color{darkblack}u}_j^{-1} \leq {\color{darkblack}\pi}, \; \forall {\color{coral}a}_{ij} \neq 0 , \\
                    &   {\color{darkblack}\pi}, \, {\color{darkblack}\psi}, {\color{darkblack}u}, \text{positive} \\
  \text{variables}  &   {\color{darkblack}\pi}, {\color{darkblack}\psi}, {\color{darkblack}u} \, .
  \end{array}$$

---

### 🔢 Optimal Matrix Scaling (cont'd)

By taking logarithms of variables, the above problem can be transformed
into:

$$\begin{array}{ll}
  \text{minimize}   &   {\color{darkblack}\pi}' - {\color{darkblack}\psi}'  \\
  \text{subject to} &   {\color{darkblack}u}_i' -  {\color{darkblack}u}_j'  \leq {\color{darkblack}\pi}' - {\color{coral}a}_{ij}', \; \forall {\color{coral}a}_{ij} \neq 0 \,, \\
                    &   {\color{darkblack}u}_j' -  {\color{darkblack}u}_i' \leq {\color{coral}a}_{ij}' - {\color{darkblack}\psi}', \; \forall {\color{coral}a}_{ij} \neq 0 \,, \\
  \text{variables}  &   {\color{darkblack}\pi}', {\color{darkblack}\psi}', {\color{darkblack}u}' \, .
  \end{array}$$

where ${\color{coral}k}'$ denotes $\log( | {\color{coral}k} | )$ and
${\color{darkblack}x} = ({\color{darkblack}\pi}', {\color{darkblack}\psi}' )^{ \mathrm{ T } }$.

---

### 📚 Example: clock period & yield-driven co-optimization

$$\begin{array}{cll}
   \text{minimize} &{\color{darkblack}T_\text{CP}} - {\color{coral}w_{\beta}}{\color{darkblack}\beta} \\
   \text{subject to} & {\color{darkblack}u_i} - {\color{darkblack}u_j} \le {\color{darkblack}T_\text{CP}} + {\color{purple}F}_{ij}^{-1}(1-{\color{darkblack}\beta}), & \forall (i,j) \in {\color{lime}E}_s \,,\\
                     & {\color{darkblack}u_j} - {\color{darkblack}u_i} \le {\color{purple}F}_{ij}^{-1}(1-{\color{darkblack}\beta}), & \forall (j,i) \in {\color{lime}E}_h \,, \\
                     & {\color{darkblack}T_\text{CP}} \ge 0, \, 0 \le {\color{darkblack}\beta} \le 1 \, , \\
    \text{variables} &{\color{darkblack}T_\text{CP}},{\color{darkblack}\beta}, {\color{darkblack}u}.
   \end{array}$$

- ❗Note that ${\color{purple}F}_{ij}^{-1}({\color{darkblack}x})$ is not concave in general in $[0,1]$.
- Fortunately, we are most likely interested in optimizing circuits
    for high yield rather than the low one in practice.
- Therefore, by imposing an additional constraint to ${\color{darkblack}\beta}$, say
    ${\color{darkblack}\beta} \geq 0.8$, the problem becomes convex.

---

### 📈 Inverse CDF

![img](lec06.files/Fig2-b-invcdf.svg)

---

Matrix Inequalities
===================

---

### 🧮 Problems With Matrix Inequalities

Consider the following problem:

$$\begin{array}{ll}
    \text{minimize}    & {\color{purple}f_0}({\color{darkblack}x}), \\
    \text{subject to}  & {\color{purple}F}({\color{darkblack}x}) \succeq 0,
\end{array}$$

- ${\color{purple}F}({\color{darkblack}x})$: a matrix-valued function
- ${\color{coral}A} \succeq 0$ denotes ${\color{coral}A}$ is positive semidefinite.

---

### 🧮 Problems With Matrix Inequalities

- Recall that a matrix ${\color{coral}A}$ is positive semidefinite if and only if
    ${\color{darkblack}v}^\mathsf{T} {\color{coral}A} {\color{darkblack}v} \ge 0$ for all ${\color{darkblack}v} \in \mathbb{R}^n$.
- The problem can be transformed into:
    $$\begin{array}{ll}
            \text{minimize}      & {\color{purple}f_0}({\color{darkblack}x}), \\
            \text{subject to}    & {\color{darkblack}v}^\mathsf{T} {\color{purple}F}({\color{darkblack}x}) {\color{darkblack}v} \ge 0, \; \forall {\color{darkblack}v} \in \mathbb{R}^n
    \end{array}$$
- Consider ${\color{darkblack}v}^\mathsf{T} {\color{purple}F}({\color{darkblack}x}) {\color{darkblack}v}$ is concave for all ${\color{darkblack}v} \in \mathbb{R}^n$ w.r.t.
    ${\color{darkblack}x}$, then the above problem is a convex programming.
- Reduce to *semidefinite programming* if ${\color{purple}f_0}({\color{darkblack}x})$ and ${\color{purple}F}({\color{darkblack}x})$
    are linear, i.e., ${\color{purple}F}({\color{darkblack}x}) = {\color{coral}F_0} + {\color{darkblack}x}_1 {\color{coral}F_1} + \cdots + {\color{darkblack}x}_m {\color{coral}F_m}$

---

### 🔮 Oracle in Matrix Inequalities

The oracle only needs to:

- Perform a *row-based* Cholesky factorization such that
    ${\color{purple}F}({\color{darkblack}x}_0) = {\color{coral}R}^\mathsf{T} {\color{coral}R}$.
- Let ${\color{coral}A}_{:p,:p}$ denotes a submatrix
    ${\color{coral}A}(1:p, 1:p) \in \mathbb{R}^{p\times p}$.
- If Cholesky factorization fails at row $p$,
  - there exists a vector
        ${\color{darkblack}e}_p = (0, 0, \cdots, 0, 1)^\mathsf{T} \in \mathbb{R}^p$, such that
    - ${\color{darkblack}v} = {\color{coral}R}_{:p,:p}^{-1} {\color{darkblack}e}_p$, and
    - ${\color{darkblack}v}^\mathsf{T} {\color{purple}F}_{:p,:p}({\color{darkblack}x}_0) {\color{darkblack}v} < 0$.
  - The cut $({\color{darkcyan} g},{\color{green} h})$ =
        $(-{\color{darkblack}v}^\mathsf{T} \partial {\color{purple}F}_{:p,:p}({\color{darkblack}x}_0) {\color{darkblack}v}, -{\color{darkblack}v}^\mathsf{T} {\color{purple}F}_{:p,:p}({\color{darkblack}x}_0) {\color{darkblack}v})$

---

### 🐍 Python code

```python
import numpy as np
from chol_ext import *

class lmi_oracle:
    def __init__(F):
        self.F = F

def chk_mtx(self, A, x):
        n = len(x)
        g = np.zeros(n)
        fj = -1.0
        v = 0.0
        for i in range(n):
            A += self.F[i] * x[i]
        R, p = chol_ext(A)
        if p == 0: return g, fj, R, v
        v = witness(R, p)
        fj = -np.dot(v, A[:p, :p].dot(v))
        for i in range(n):
            g[i] = -np.dot(v, self.F[i][:p, :p].dot(v))
        return g, fj, R, v
```

---

### 📚 Example: Matrix Norm Minimization

- Let ${\color{purple}A}({\color{darkblack}x}) = {\color{coral}A_0} + {\color{darkblack}x}_1 {\color{coral}A_1} + \cdots + {\color{darkblack}x}_m {\color{coral}A_m}$
- Problem $\min_{{\color{darkblack}x}} \| {\color{purple}A}({\color{darkblack}x}) \|$ can be reformulated as
 $$\begin{array}{ll}
       \text{minimize}      & {\color{coral}t}, \\
       \text{subject to}    & \left(
   \begin{array}{cc}
    {\color{coral}t}\,I   & {\color{purple}A}({\color{darkblack}x}) \\
    {\color{purple}A}^\mathsf{T}({\color{darkblack}x}) & {\color{coral}t}\,I
   \end{array} \right) \succeq 0,
 \end{array}$$
- Binary search on ${\color{coral}t}$ can be used for this problem.

---

### 📚 Example: Estimation of Correlation Function

$$\begin{array}{ll}
   \min_{{\color{darkmagenta}\kappa}, {\color{darkblack}p}}   & \| {\color{red}\Omega}({\color{darkblack}p}) + {\color{darkmagenta}\kappa} I - {\color{red}Y} \| \\
   \text{s. t.} & {\color{red}\Omega}({\color{darkblack}p}) \succcurlyeq 0,  {\color{darkmagenta}\kappa} \geq 0 \; .\\
 \end{array}$$

- Let ${\color{purple}\rho}({\color{brown}h}) = \sum_i^m {\color{darkblack}p}_i {\color{purple}\Psi_i}({\color{brown}h})$, where
  - ${\color{darkblack}p}_i$'s are the unknown coefficients to be fitted
  - ${\color{purple}\Psi_i}$'s are a family of basis functions.
- The covariance matrix ${\color{red}\Omega}({\color{darkblack}p})$ can be recast as:

$${\color{red}\Omega}({\color{darkblack}p}) = {\color{darkblack}p}_1 {\color{red}F_1} + \cdots + {\color{darkblack}p}_m {\color{red}F_m}$$

where $\{{\color{red}F_k}\}_{i,j} = {\color{purple}\Psi_k}( \| {\color{darkblack}s_j} - {\color{darkblack}s_i} \|_2)$

---

## Q&A 🎤