layout: true class: typo, typo-selection --- count: false class: nord-dark, middle, center Lecture 2c: Introduction to Convex Optimization 🎯 ======================================= .pull-left[ @luk036 👨‍💻 · 2026 📅 ] .pull-right[  ] --- ### 📋 Agenda .pull-left[ **Foundations** 🎯 - Introduction 🚀 - Linear programming 🧮 - Nonlinear programming 📈 - Duality and Convexity ⚖️ ] .pull-right[ **Techniques** 🛠️ - Approximation techniques 🛠️ - Convex Optimization 🎯 - Books and online resources 📚 ] --- Classification of Optimizations 🔍 ------------------------------- - Continuous - Linear vs Non-linear - Convex vs Non-convex - Discrete - Polynomial time Solvable - NP-hard - Approximatable - Non-approximatable - Mixed --- Continuous Optimization 📊 -----------------------  --- Linear Programming Problem 🧮 -------------------------- - An LPP in standard form is: $$\min\{ {\color{darkcyan} c^\mathsf{T} } {\color{darkblack}x} \mid {\color{red} A} {\color{darkblack}x} = {\color{darkcyan} b}, {\color{darkblack}x} \ge 0\}.$$ - The ingredients of LPP are: - An ${\color{coral} m} \times {\color{coral} n}$ matrix ${\color{red} A}$, with ${\color{coral} n} > {\color{coral} m}$ - A vector ${\color{darkcyan} b} \in \mathbb{R}^{\color{coral} m}$ - A vector ${\color{darkcyan} c} \in \mathbb{R}^{\color{coral} n}$ --- 📚 Example ------- $$\begin{array}{lll} \text{minimize} & 0.4 {\color{darkblack}x_1} + 3.4 {\color{darkblack}x_2} - 3.4 {\color{darkblack}x_3} \\ \text{subject to} & 0.5 {\color{darkblack}x_1} + 0.5 {\color{darkblack}x_2} & = 3.5 \\ & 0.3 {\color{darkblack}x_1} - 0.8 {\color{darkblack}x_2} + 8.4 {\color{darkblack}x_2} & = 4.5 \\ & {\color{darkblack}x_1}, {\color{darkblack}x_2}, {\color{darkblack}x_3} \ge 0 \end{array}$$ --- Transformations to Standard Form 🔄 -------------------------------- - Theorem: Any LPP can be transformed into the standard form. - Variables not restricted in sign: - Decompose ${\color{darkblack}x}$ to two new variables ${\color{darkblack}x} = {\color{darkblack}x_1} - {\color{darkblack}x_2}, {\color{darkblack}x_1}, {\color{darkblack}x_2} \geq 0$ - Transforming inequalities into equalities: - By putting slack variable ${\color{darkcyan} y} = {\color{darkcyan} b} - {\color{red} A} {\color{darkblack}x} \geq 0$ - Set ${\color{darkblack}x'} = ({\color{darkblack}x}, {\color{darkcyan} y}), {\color{red} A'} = ({\color{red} A}, 1)$ - Transforming a max into a min - max(expression) = min($-$expression); --- Duality of LPP ⚖️ -------------- - If the primal problem of the LPP: $\min\{ {\color{darkcyan} c^\mathsf{T} } {\color{darkblack}x} \mid {\color{red} A} {\color{darkblack}x} \ge {\color{darkcyan} b}, {\color{darkblack}x} \ge 0\}$. - Its dual is: $\max\{ {\color{darkcyan} y}^\mathsf{T} {\color{darkcyan} b} \mid {\color{red} A}^\mathsf{T} {\color{darkcyan} y} \leq {\color{darkcyan} c}, {\color{darkcyan} y} \ge 0\}$. - If the primal problem is: $\min\{ {\color{darkcyan} c}^\mathsf{T} {\color{darkblack}x} \mid {\color{red} A} {\color{darkblack}x} = {\color{darkcyan} b}, {\color{darkblack}x} \ge 0\}$. - Its dual is: $\max\{ {\color{darkcyan} y}^\mathsf{T} {\color{darkcyan} b} \mid {\color{red} A}^\mathsf{T} {\color{darkcyan} y} \leq {\color{darkcyan} c}\}$. --- Nonlinear Programming 📈 --------------------- - The standard form of an NLPP is $$\min\{{\color{purple}f}({\color{darkblack}x}) \mid {\color{purple}g}({\color{darkblack}x}) \leq 0, {\color{green}h}({\color{darkblack}x})=0 \}.$$ - Necessary conditions of optimality, Karush- Kuhn-Tucker (KKT) conditions: - $\nabla {\color{purple}f}({\color{darkblack}x}) + {\color{firebrick} \mu} \nabla {\color{purple}g}({\color{darkblack}x}) + {\color{firebrick} \lambda} \nabla {\color{green}h}({\color{darkblack}x}) = 0$, - ${\color{firebrick} \mu} {\color{purple}g}({\color{darkblack}x}) = 0$, - ${\color{firebrick} \mu} \geq 0, {\color{purple}g}({\color{darkblack}x}) \leq 0, {\color{green}h}({\color{darkblack}x}) = 0$ --- Convexity 🏔️ --------- - A function ${\color{purple}f}$: ${\color{steelblue}K} \subseteq \mathbb{R}^{\color{coral} n} \mapsto R$ is convex if ${\color{steelblue}K}$ is a convex set and ${\color{purple}f}({\color{darkcyan} y}) \ge {\color{purple}f}({\color{darkblack} x}) + \nabla {\color{purple}f}({\color{darkblack} x}) ({\color{darkcyan} y} - {\color{darkblack} x}), \; {\color{darkcyan} y},{\color{darkblack} x} \in {\color{steelblue}K}$. - **Theorem**: Assume that ${\color{purple}f}$ and ${\color{purple}g}$ are convex differentiable functions. If the pair $({\color{darkblack}x}, {\color{firebrick}m})$ satisfies the KKT conditions above, ${\color{darkblack}x}$ is an optimal solution of the problem. If in addition, ${\color{purple}f}$ is strictly convex, ${\color{darkblack}x}$ is the only solution of the problem. **(Local minimum = global minimum)** --- Convex vs Non-convex Optimization 🔀 ---------------------------------- .mermaid[ <pre> graph TD A[Optimization Problem] --> B{Convex?} B -->|Yes| C["Global Optimum\n Guaranteed"] B -->|No| D["Local Optimum\n Only"] C --> E["Efficient\n Algorithms"] D --> F["Heuristic\n Methods"] style A fill:#e3f2fd,stroke:#1565c0,stroke-width:3px style B fill:#ffcdd2,stroke:#c62828,stroke-width:3px style C fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px style D fill:#fff3e0,stroke:#e65100,stroke-width:3px style E fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px style F fill:#fff3e0,stroke:#e65100,stroke-width:3px </pre> ] --- Duality and Convexity ⚖️ --------------------- - Dual is the NLPP: $$\max\{ {\color{purple}\theta}({\color{firebrick} \mu}, {\color{firebrick} \lambda}) \mid {\color{firebrick} \mu} \geq 0\},$$ where ${\color{purple}\theta}({\color{firebrick} \mu}, {\color{firebrick} \lambda}) = \inf_{\color{darkblack}x} [ {\color{purple}f}({\color{darkblack}x}) + {\color{firebrick} \mu} {\color{purple}g}({\color{darkblack}x}) + {\color{firebrick} \lambda} {\color{green}h}({\color{darkblack}x}) ]$ - Dual problem is always convex. - Useful for computing the lower/upper bound. --- class: nord-light, middle, center Convexify the non-convex's 🔄 ========================== --- Change of curvature: square 🔄 --------------------------- Transform: $$0.3 \leq {\color{red} \sqrt{x} } \leq 0.4$$ into: $$0.09 \leq {\color{green} x} \leq 0.16 \, .$$ 👉 Note that $\sqrt{\cdot}$ are **monotonic** **concave** functions in $(0, +\infty)$. Generalization: - Consider $|H({\color{green}\omega})|^2$ (power) instead of $|H({\color{green}\omega})|$ (magnitude). - square root -> Spectral factorization --- Change of curvature: square 🔄 --------------------------- Transform: $${\color{red} x^2 } + {\color{red} y^2 } \geq 0.16, \quad \text{(non-convex)}$$ into: $${\color{green} x'} + {\color{green} y'} \geq 0.16, \quad {\color{green} x'}, {\color{green} y'} \geq 0$$ Then: $${\color{red} x}_\text{opt} = \pm\sqrt{{\color{green} x'}_\text{opt} }, \quad {\color{red} y}_\text{opt} = \pm\sqrt{{\color{green} y'}_\text{opt} }.$$ --- Change of curvature: sine 📐 --------------------------- Transform: $${\color{red} \sin{x} } \leq 0.4, \quad 0 \leq {\color{red} x} \leq \pi/2$$ into: $${\color{green} y} \leq 0.4, \quad 0 \leq {\color{green} y} \leq 1$$ Then: $${\color{red} x}_\text{opt} = \sin^{-1}({\color{green} y}_\text{opt}).$$ 👉 Note that $\sin(\cdot)$ are monotonic concave functions in $(0, \pi/2)$. --- Change of curvature: log 📐 --------------------------- Transform: $${\color{darkcyan}\pi} \leq {\color{red} x / y} \leq {\color{darkcyan}\phi}$$ into: $${\color{darkcyan}\pi'} \leq {\color{green} x' - y'} \leq {\color{darkcyan}\phi'}$$ where ${\color{green} z'} = \log({\color{red} z})$. Then: $${\color{red} z}_\text{opt} = \exp({\color{green} z'}_\text{opt}).$$ Generalization: - Geometric programming --- Change of curvature: inverse 🔄 --------------------------- Transform: $${\color{red} \log(x)} + 0.4 \leq 0, \; {\color{red} x} > 0$$ into: $${\color{green} -\log(y)} + 0.4 \leq 0, \; {\color{green} y} > 0 \, .$$ Then: $${\color{red} x}_\text{opt} = {\color{green} y}^{-1}_\text{opt}.$$ 👉 Note that $\sqrt{\cdot}$, $\log(\cdot)$, and $(\cdot)^{-1}$ are monotonic functions. --- Generalize to matrix inequalities 📊 --------------------------------- Transform: $${\color{red} \log(\det X)} + \text{Tr}({\color{red} X}^{-1} {\color{darkcyan}C}) \leq 0.3, \; {\color{red} X} \succ 0$$ into: $${\color{green} -\log(\det Y)} + \text{Tr}({\color{green} Y} {\color{darkcyan}C}) \leq 0.3, \; {\color{green} Y} \succ 0$$ Then: $${\color{red} X}_\text{opt} = {\color{green} Y}^{-1}_\text{opt}.$$ --- Change of variables 🔀 ---------------------- Transform: $$({\color{darkcyan}a} + {\color{darkcyan}b} {\color{red} y}) {\color{darkblack}x} \leq 0, \; {\color{darkblack}x} > 0$$ into: $${\color{darkcyan}a} {\color{darkblack}x} + {\color{darkcyan}b} {\color{green} z} \leq 0, \; {\color{darkblack}x} > 0$$ where ${\color{green} z} = {\color{red} y} {\color{darkblack}x}$. Then: $${\color{red} y}_\text{opt} = {\color{green} z}_\text{opt} {\color{darkblack}x}^{-1}_\text{opt}$$ --- Generalize to matrix inequalities 📊 --------------------------------- Transform: $$({\color{darkcyan}A} + {\color{darkcyan}B} {\color{red} Y}) {\color{darkblack}X} + {\color{darkblack}X} ({\color{darkcyan}A} + {\color{darkcyan}B} {\color{red} Y})^T \prec 0, \; {\color{darkblack}X} \succ 0$$ into: $${\color{darkcyan}A} {\color{darkblack}X} + {\color{darkblack}X} {\color{darkcyan}A}^T + {\color{darkcyan}B} {\color{green} Z} + {\color{green} Z}^T {\color{darkcyan}B}^T \prec 0, \; {\color{darkblack}X} \succ 0$$ where ${\color{green} Z} = {\color{red} Y} {\color{darkblack}X}$. Then: $${\color{red} Y}_\text{opt} = {\color{green} Z}_\text{opt} {\color{darkblack}X}^{-1}_\text{opt}$$ --- Other thoughts 🧠 --------------- - Minimizing any quasi-convex function subject to convex constraints can easily be transformed into a convex programming. - Replace a non-convex constraint with a sufficient condition (such as its lower bound). Less optimal. - Relaxation + heuristic - Decomposition --- Unconstraint Techniques ⚙️ ----------------------- - Line search methods - Fixed or variable step size - Interpolation - Golden section method - Fibonacci's method - Gradient methods - Steepest descent - Quasi-Newton methods - Conjugate Gradient methods --- General Descent Method 📉 ---------------------- 1. **Input**: a starting point ${\color{darkblack}x} \in$ dom ${\color{purple}f}$ 2. **Output**: ${\color{darkblack}x}^*$ 3. **repeat** 1. Determine a descent direction ${\color{darkcyan} p}$. 2. Line search. Choose a step size ${\color{green} \alpha} > 0$. 3. Update. ${\color{darkblack}x} := {\color{darkblack}x} + {\color{green} \alpha} {\color{darkcyan} p}$ 4. **until** stopping criterion satisfied. --- Some Common Descent Directions 🧭 ------------------------------ - Gradient descent: ${\color{darkcyan} p} = -\nabla {\color{purple}f}({\color{darkblack}x})^\mathsf{T}$ - Steepest descent: - $\triangle {\color{darkblack}x}_{nsd}$ = argmin$\{\nabla {\color{purple}f}({\color{darkblack}x})^\mathsf{T} {\color{darkblack}v} \mid \|{\color{darkblack}v}\|=1 \}$ - $\triangle {\color{darkblack}x}$ = $\|\nabla {\color{purple}f}({\color{darkblack}x})\| \triangle {\color{darkblack}x}_{nsd}$ (un-normalized) - Newton's method: - ${\color{darkcyan} p} = -\nabla^2 {\color{purple}f}({\color{darkblack}x})^{-1} \nabla {\color{purple}f}({\color{darkblack}x})$ - Conjugate gradient method: - ${\color{darkcyan} p}$ is "orthogonal" to all previous ${\color{darkcyan} p}$'s - Stochastic subgradient method: - ${\color{darkcyan} p}$ is calculated from a set of sample data (instead of using all data) - Network flow problems: - ${\color{darkcyan} p}$ is given by a "negative cycle" (or "negative cut"). --- Approximation Under Constraints 🛠️ ------------------------------- - Penalization and barriers - Dual method - Interior Point method - Augmented Lagrangian method --- 📚 Books and Online Resources ---------------------------- - Pablo Pedregal. Introduction to Optimization, Springer. 2003 (O224 P371) - Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Dec. 2002 - Mittlemann, H. D. and Spellucci, P. Decision Tree for Optimization Software, 2003 --- count: false class: nord-dark, middle, center .pull-left[ # Q&A 🎤 ] .pull-right[  ]