layout: true class: typo, typo-selection --- count: false class: nord-dark, middle, center Lecture 2c: Introduction to Convex Optimization ======================================= .pull-left[ @luk036 👨‍💻 · 2026 📅 ] .pull-right[  ] --- Overview 📋 ------ - Introduction - Linear programming - Nonlinear programming - Duality and Convexity - Approximation techniques - Convex Optimization - Books and online resources. --- Classification of Optimizations ------------------------------- - Continuous - Linear vs Non-linear - Convex vs Non-convex - Discrete - Polynomial time Solvable - NP-hard - Approximatable - Non-approximatable - Mixed --- Continuous Optimization -----------------------  --- Linear Programming Problem -------------------------- - An LPP in standard form is: $$\min\{ c^\mathsf{T} x \mid A x = b, x \ge 0\}.$$ - The ingredients of LPP are: - An $m \times n$ matrix $A$, with $n > m$ - A vector $b \in \mathbb{R}^m$ - A vector $c \in \mathbb{R}^n$ --- 📚 Example ------- $$\begin{array}{lll} \text{minimize} & 0.4 x_1 + 3.4 x_2 - 3.4 x_3 \\ \text{subject to} & 0.5 x_1 + 0.5 x_2 & = 3.5 \\ & 0.3 x_1 - 0.8 x_2 + 8.4 x_2 & = 4.5 \\ & x_1, x_2, x_3 \ge 0 \end{array}$$ --- Transformations to Standard Form -------------------------------- - Theorem: Any LPP can be transformed into the standard form. - Variables not restricted in sign: - Decompose $x$ to two new variables $x = x_1 - x_2, x_1, x_2 \geq 0$ - Transforming inequalities into equalities: - By putting slack variable $y = b - A x \geq 0$ - Set $x' = (x, y), A' = (A, 1)$ - Transforming a max into a min - max(expression) = min($-$expression); --- Duality of LPP -------------- - If the primal problem of the LPP: $\min\{ c^\mathsf{T} x \mid A x \ge b, x \ge 0\}$. - Its dual is: $\max\{ y^\mathsf{T} b \mid A^\mathsf{T} y \leq c, y \ge 0\}$. - If the primal problem is: $\min\{ c^\mathsf{T} x \mid A x = b, x \ge 0\}$. - Its dual is: $\max\{ y^\mathsf{T} b \mid A^\mathsf{T} y \leq c\}$. --- Nonlinear Programming --------------------- - The standard form of an NLPP is $$\min\{f(x) \mid g(x) \leq 0, h(x)=0 \}.$$ - Necessary conditions of optimality, Karush- Kuhn-Tucker (KKT) conditions: - $\nabla f(x) + \mu \nabla g(x) + \lambda \nabla h(x) = 0$, - $\mu g(x) = 0$, - $\mu \geq 0, g(x) \leq 0, h(x) = 0$ --- Convexity --------- - A function $f$: $K \subseteq \mathbb{R}^n \mapsto R$ is convex if $K$ is a convex set and $f(y) \ge f(x) + \nabla f(x) (y - x), \; y,x \in K$. - **Theorem**: Assume that $f$ and $g$ are convex differentiable functions. If the pair $(x, m)$ satisfies the KKT conditions above, $x$ is an optimal solution of the problem. If in addition, $f$ is strictly convex, $x$ is the only solution of the problem. **(Local minimum = global minimum)** --- Convex vs Non-convex Optimization ---------------------------------- .mermaid[ <pre> graph TD A[Optimization Problem] --> B{Convex?} B -->|Yes| C["Global Optimum\n Guaranteed"] B -->|No| D["Local Optimum\n Only"] C --> E["Efficient\n Algorithms"] D --> F["Heuristic\n Methods"] style A fill:#2196f3 style B fill:#f44336 style C fill:#4caf50 style D fill:#ff9800 style E fill:#4caf50 style F fill:#ff9800 </pre> ] --- Duality and Convexity --------------------- - Dual is the NLPP: $$\max\{\theta(\mu, \lambda) \mid \mu \geq 0\},$$ where $\theta(\mu, \lambda) = \inf_x [ f(x) + \mu g(x) + \lambda h(x) ]$ - Dual problem is always convex. - Useful for computing the lower/upper bound. --- class: nord-light, middle, center Convexify the non-convex's ========================== --- Change of curvature: square --------------------------- Transform: $$0.3 \leq {\color{red} \sqrt{x} } \leq 0.4$$ into: $$0.09 \leq {\color{green} x} \leq 0.16 \, .$$ 👉 Note that $\sqrt{\cdot}$ are **monotonic** **concave** functions in $(0, +\infty)$. Generalization: - Consider $|H(\omega)|^2$ (power) instead of $|H(\omega)|$ (magnitude). - square root -> Spectral factorization --- Change of curvature: square --------------------------- Transform: $${\color{red} x^2 } + {\color{red} y^2 } \geq 0.16, \quad \text{(non-convex)}$$ into: $${\color{green} x'} + {\color{green} y'} \geq 0.16, \quad x', y' \geq 0$$ Then: $$x_\text{opt} = \pm\sqrt{x'_\text{opt} }, \quad y_\text{opt} = \pm\sqrt{y'_\text{opt} }.$$ --- Change of curvature: sine --------------------------- Transform: $${\color{red} \sin{x} } \leq 0.4, \quad 0 \leq x \leq \pi/2$$ into: $${\color{green} y} \leq 0.4, \quad 0 \leq y \leq 1$$ Then: $$x_\text{opt} = \sin^{-1}(y_\text{opt}).$$ 👉 Note that $\sin(\cdot)$ are monotonic concave functions in $(0, \pi/2)$. --- Change of curvature: log --------------------------- Transform: $$\pi \leq {\color{red} x / y} \leq \phi$$ into: $$\pi' \leq {\color{green} x' - y'} \leq \phi'$$ where $z' = \log(z)$. Then: $$z_\text{opt} = \exp(z'_\text{opt}).$$ Generalization: - Geometric programming --- Change of curvature: inverse --------------------------- Transform: $${\color{red} \log(x)} + 0.4 \leq 0, \; x > 0$$ into: $${\color{green} -\log(y)} + 0.4 \leq 0, \; y > 0 \, .$$ Then: $$x_\text{opt} = y^{-1}_\text{opt}.$$ 👉 Note that $\sqrt{\cdot}$, $\log(\cdot)$, and $(\cdot)^{-1}$ are monotonic functions. --- Generalize to matrix inequalities --------------------------------- Transform: $${\color{red} \log(\det X)} + \text{Tr}(X^{-1} C) \leq 0.3, \; X \succ 0$$ into: $${\color{green} -\log(\det Y)} + \text{Tr}(Y C) \leq 0.3, \; Y \succ 0$$ Then: $$X_\text{opt} = Y^{-1}_\text{opt}.$$ --- Change of variables ---------------------- Transform: $$(a + b {\color{red} y}) x \leq 0, \; x > 0$$ into: $$a x + b {\color{green} z} \leq 0, \; x > 0$$ where $z = y x$. Then: $$y_\text{opt} = z_\text{opt} x^{-1}_\text{opt}$$ --- Generalize to matrix inequalities --------------------------------- Transform: $$(A + B {\color{red} Y}) X + X (A + B {\color{red} Y})^T \prec 0, \; X \succ 0$$ into: $$A X + X A^T + B {\color{green} Z} + {\color{green} Z}^T B^T \prec 0, \; X \succ 0$$ where $Z = Y X$. Then: $$Y_\text{opt} = Z_\text{opt} X^{-1}_\text{opt}$$ --- Other thoughts --------------- - Minimizing any quasi-convex function subject to convex constraints can easily be transformed into a convex programming. - Replace a non-convex constraint with a sufficient condition (such as its lower bound). Less optimal. - Relaxation + heuristic - Decomposition --- Unconstraint Techniques ----------------------- - Line search methods - Fixed or variable step size - Interpolation - Golden section method - Fibonacci's method - Gradient methods - Steepest descent - Quasi-Newton methods - Conjugate Gradient methods --- General Descent Method ---------------------- 1. **Input**: a starting point $x \in$ dom $f$ 2. **Output**: $x^*$ 3. **repeat** 1. Determine a descent direction $p$. 2. Line search. Choose a step size $\alpha > 0$. 3. Update. $x := x + \alpha p$ 4. **until** stopping criterion satisfied. --- Some Common Descent Directions ------------------------------ - Gradient descent: $p = -\nabla f(x)^\mathsf{T}$ - Steepest descent: - $\triangle x_{nsd}$ = argmin$\{\nabla f(x)^\mathsf{T} v \mid \|v\|=1 \}$ - $\triangle x$ = $\|\nabla f(x)\| \triangle x_{nsd}$ (un-normalized) - Newton's method: - $p = -\nabla^2 f(x)^{-1} \nabla f(x)$ - Conjugate gradient method: - $p$ is "orthogonal" to all previous $p$'s - Stochastic subgradient method: - $p$ is calculated from a set of sample data (instead of using all data) - Network flow problems: - $p$ is given by a "negative cycle" (or "negative cut"). --- Approximation Under Constraints ------------------------------- - Penalization and barriers - Dual method - Interior Point method - Augmented Lagrangian method --- 📚 Books and Online Resources ---------------------------- - Pablo Pedregal. Introduction to Optimization, Springer. 2003 (O224 P371) - Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Dec. 2002 - Mittlemann, H. D. and Spellucci, P. Decision Tree for Optimization Software, 2003 --- count: false class: nord-dark, middle, center .pull-left[ # Q&A 🎤 ] .pull-right[  ]