Convex optimization

Convex optimization is a subfield of mathematical optimization that studies the problem of minimizing convex functions over convex sets (or, equivalently, maximizing concave functions over convex sets). Many classes of convex optimization problems admit polynomial-time algorithms, whereas mathematical optimization is in general NP-hard.

Convex optimization is a subfield of mathematical optimization that studies the problem of minimizing convex functions over convex sets (or, equivalently, maximizing concave functions over convex sets). Many classes of convex optimization problems admit polynomial-time algorithms, whereas mathematical optimization is in general NP-hard.

Definition

Standard form

A convex optimization problem is in standard form if it is written as \begin{align} &\underset{\mathbf{x}}{\operatorname{minimize}}& & f(\mathbf{x}) \\ &\operatorname{subject\ to} & &g_i(\mathbf{x}) \leq 0, \quad i = 1, \dots, m \\ &&&h_i(\mathbf{x}) = 0, \quad i = 1, \dots, p, \end{align} where: \mathbf{x} \in \mathbb{R}^n is the vector of optimization variables; The objective function f: \mathcal D \subseteq \mathbb{R}^n \to \mathbb{R} is a convex function; The inequality constraint functions g_i : \mathbb{R}^n \to \mathbb{R} , i=1, \ldots, m , are convex functions; The equality constraint functions h_i : \mathbb{R}^n \to \mathbb{R} , i=1, \ldots, p , are affine transformations, that is, of the form: h_i(\mathbf{x}) = \mathbf{a_i}\cdot \mathbf{x} - b_i , where \mathbf{a_i} is a vector and b_i is a scalar. The feasible set C of the optimization problem consists of all points \mathbf{x} \in \mathcal{D} satisfying the inequality and the equality constraints. This set is convex because \mathcal{D} is convex, the sublevel sets of convex functions are convex, affine sets are convex, and the intersection of convex sets is convex. Many optimization problems can be equivalently formulated in this standard form. For example, the problem of maximizing a concave function f can be re-formulated equivalently as the problem of minimizing the convex function -f . The problem of maximizing a concave function over a convex set is commonly called a convex optimization problem.

Epigraph form (standard form with linear objective)

In the standard form it is possible to assume, without loss of generality, that the objective function f is a linear function. This is because any program with a general objective can be transformed into a program with a linear objective by adding a single variable t and a single constraint, as follows: \begin{align} &\underset{\mathbf{x},t}{\operatorname{minimize}}& & t \\ &\operatorname{subject\ to} & &f(\mathbf{x}) - t \leq 0 \\ && &g_i(\mathbf{x}) \leq 0, \quad i = 1, \dots, m \\ &&&h_i(\mathbf{x}) = 0, \quad i = 1, \dots, p, \end{align} It is possible to convert a convex program in standard form, to a convex program with no equality constraints. Denote the equality constraints h i (x)=0 as Ax=b, where A has n columns. If Ax=b is infeasible, then of course the original problem is infeasible. Otherwise, it has some solution x 0 , and the set of all solutions can be presented as: Fz+x 0 , where z is in R k , k=n-rank(A), and F is an n-by-k matrix. Substituting x = Fz+x 0 in the original problem gives: \begin{align} &\underset{\mathbf{x}}{\operatorname{minimize}}& & f(\mathbf{F \mathbf{z} + \mathbf{x}_0}) \\ &\operatorname{subject\ to} & &g_i(\mathbf{F \mathbf{z} + \mathbf{x}_0}) \leq 0, \quad i = 1, \dots, m \\ \end{align} where the variables are z. Note that there are rank(A) fewer variables. This means that, in principle, one can restrict attention to convex optimization problems without equality constraints. In practice, however, it is often preferred to retain the equality constraints, since they might make some algorithms more efficient, and also make the problem easier to understand and analyze.

Special cases

The following problem classes are all convex optimization problems, or can be reduced to convex optimization problems via simple transformations: , QP: quadratic programming, SOCP second-order cone program, SDP: semidefinite programming, CP: conic optimization.)]] Linear programming problems are the simplest convex programs. In LP, the objective and constraint functions are all linear. Quadratic programming are the next-simplest. In QP, the constraints are all linear, but the objective may be a convex quadratic function. Second order cone programming are more general. Semidefinite programming are more general. Conic optimization are even more general - see figure to the right, Other special cases include; Least squares Quadratic minimization with convex quadratic constraints Geometric programming Entropy maximization with appropriate constraints.

Properties

The following are useful properties of convex optimization problems: every point that is local minimum is also a global minimum; the optimal set is convex; if the objective function is strictly convex, then the problem has at most one optimal point. These results are used by the theory of convex minimization along with geometric notions from functional analysis (in Hilbert spaces) such as the Hilbert projection theorem, the separating hyperplane theorem, and Farkas' lemma.

Algorithms

\mathcal{X} = \left\{x\in X \vert g_1(x), \ldots, g_m(x)\leq 0\right\}. The Lagrangian function for the problem is L(x,\lambda_{0},\lambda_1, \ldots ,\lambda_{m})=\lambda_{0} f(x) + \lambda_{1} g_{1} (x)+\cdots + \lambda_{m} g_{m} (x). For each point x in X that minimizes f over X , there exist real numbers \lambda_{0},\lambda_1, \ldots, \lambda_{m}, called Lagrange multipliers, that satisfy these conditions simultaneously: x minimizes L(y,\lambda_{0},\lambda_{1},\ldots ,\lambda_{m}) over all y \in X, \lambda_{0},\lambda_{1},\ldots ,\lambda_{m} \geq 0, with at least one \lambda_{k} > 0, \lambda_{1}g_{1}(x)=\cdots = \lambda_{m}g_{m}(x) = 0 (complementary slackness). If there exists a "strictly feasible point", that is, a point z satisfying g_{1}(z), \ldots, g_{m}(z) then the statement above can be strengthened to require that \lambda_{0}=1 . Conversely, if some x in X satisfies (1)–(3) for scalars \lambda_{0},\ldots,\lambda_{m} with \lambda_{0}=1 then x is certain to minimize f over X .

Extensions

Extensions of convex optimization include the optimization of biconvex, pseudo-convex, and quasiconvex functions. Extensions of the theory of convex analysis and iterative methods for approximately solving non-convex minimization problems occur in the field of generalized convexity, also known as abstract convex analysis.

See also

Duality Karush–Kuhn–Tucker conditions Optimization problem Proximal gradient method Algorithmic problems on convex sets

Notes

References

Hiriart-Urruty, Jean-Baptiste, and Lemaréchal, Claude. (2004). Fundamentals of Convex analysis. Berlin: Springer. Nesterov, Yurii. (2004). Introductory Lectures on Convex Optimization, Kluwer Academic Publishers Schmit, L.A.; Fleury, C. 1980: Structural synthesis by combining approximation concepts and dual methods. J. Amer. Inst. Aeronaut. Astronaut 18, 1252-1260

External links

EE364a: Convex Optimization I and EE364b: Convex Optimization II, Stanford course homepages 6.253: Convex Analysis and Optimization, an MIT OCW course homepage Brian Borchers, An overview of software for convex optimization Convex Optimization Book by Lieven Vandenberghe and Stephen P. Boyd

Text derived from https://en.wikipedia.org/w/index.php?oldid=1375043885 under CC BY-SA 4.0; modifications have been made.