Duality (optimization)

Uses your system voice; voice and rate in settings.

Snapshot

≈ 9 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

In mathematical optimization theory, duality or the duality principle is the principle that optimization problems may be viewed from either of two perspectives, the primal problem or the dual problem. If the primal is a minimization problem then the dual is a maximization problem (and vice versa). Any feasible solution to the primal (minimization) problem is at least as large as any feasible solution to the dual (maximization) problem. Therefore, the solution to the primal is an upper bound to the solution of the dual, and the solution of the dual is a lower bound to the solution of the primal. This fact is called weak duality. In general, the optimal values of the primal and dual problems need not be equal. Their difference is called the duality gap. For convex optimization problems, the duality gap is zero under a constraint qualification condition. This fact is called strong duality.

In mathematical optimization theory, duality or the duality principle is the principle that optimization problems may be viewed from either of two perspectives, the primal problem or the dual problem. If the primal is a minimization problem then the dual is a maximization problem (and vice versa). Any feasible solution to the primal (minimization) problem is at least as large as any feasible solution to the dual (maximization) problem. Therefore, the solution to the primal is an upper bound to the solution of the dual, and the solution of the dual is a lower bound to the solution of the primal. This fact is called weak duality. In general, the optimal values of the primal and dual problems need not be equal. Their difference is called the duality gap. For convex optimization problems, the duality gap is zero under a constraint qualification condition. This fact is called strong duality.

Dual problem

Usually the term "dual problem" refers to the Lagrangian dual problem but other dual problems are used – for example, the Wolfe dual problem and the Fenchel dual problem. The Lagrangian dual problem is obtained by forming the Lagrangian of a minimization problem by using nonnegative Lagrange multipliers to add the constraints to the objective function, and then solving for the primal variable values that minimize the original objective function. This solution gives the primal variables as functions of the Lagrange multipliers, which are called dual variables, so that the new problem is to maximize the objective function with respect to the dual variables under the derived constraints on the dual variables (including at least the nonnegativity constraints). In general given two dual pairs of separated locally convex spaces \left(X,X^*\right) and \left(Y,Y^*\right) and the function f: X \to \mathbb{R} \cup \{+\infty\} , we can define the primal problem as finding \hat{x} such that f(\hat{x}) = \inf_{x \in X} f(x). \, In other words, if \hat{x} exists, f(\hat{x}) is the minimum of the function f and the infimum (greatest lower bound) of the function is attained. If there are constraint conditions, these can be built into the function f by letting \tilde{f} = f + I_{\mathrm{constraints}} where I_{\mathrm{constraints}} is a suitable function on X that has a minimum 0 on the constraints, and for which one can prove that \inf_{x \in X} \tilde{f}(x) = \inf_{x \ \mathrm{constrained}} f(x) . The latter condition is trivially, but not always conveniently, satisfied for the characteristic function (i.e. I_{\mathrm{constraints}}(x) = 0 for x satisfying the constraints and I_{\mathrm{constraints}}(x) = \infty otherwise). Then extend \tilde{f} to a perturbation function F: X \times Y \to \mathbb{R} \cup \{+\infty\} such that F(x,0) = \tilde{f}(x) . The duality gap is the difference of the right and left hand sides of the inequality \sup_{y^* \in Y^*} -F^*(0,y^*) \le \inf_{x \in X} F(x,0), \, where F^* is the convex conjugate in both variables and \sup denotes the supremum (least upper bound). \end{align} The problem has constraints; we would like to convert it to a program without constraints. Theoretically, it is possible to do it by minimizing the function J(x) , defined as J(x) = f_0(x) + \sum_i I[f_i(x)] where I is an infinite step function: I[u]=0 if u \leq 0 , and I[u]=\infty otherwise. But J(x) is hard to solve as it is not continuous. It is possible to "approximate" I[u] by \lambda u , where \lambda is a positive constant. This yields a function known as the Lagrangian: L(x,\lambda) = f_0(x) + \sum_i \lambda_i f_i(x) Note that, for every x , \max_{\lambda\geq 0} L(x,\lambda) = J(x) . Proof: If x satisfies all constraints f_i(x) \leq 0 , then L(x, \lambda) is maximized when taking \lambda = 0 , and its value is then f(x) ; If x violates some constraint, f_i(x) > 0 for some i , then L(x, \lambda) \to \infty when \lambda_i \to \infty . Therefore, the original problem is equivalent to: \min_x \max_{\lambda\geq 0} L(x,\lambda) . By reversing the order of min and max, we get: \max_{\lambda\geq 0} \min_x L(x,\lambda) . The dual function is the inner problem in the above formula: g(\lambda) := \min_x L(x,\lambda) . The Lagrangian dual program is the program of maximizing g: \max_{\lambda\geq 0} g(\lambda) . The optimal solution to the dual program is a lower bound for the optimal solution of the original (primal) program; this is the weak duality principle. If the primal problem is convex and bounded from below, and there exists a point in which all nonlinear constraints are strictly satisfied (Slater's condition), then the optimal solution to the dual program equals the optimal solution of the primal program; this is the strong duality principle. In this case, we can solve the primal program by finding an optimal solution \lambda^* to the dual program, and then solving: \min_x L(x,\lambda^*) . Note that, to use either the weak or the strong duality principle, we need a way to compute g(\lambda) . In general this may be hard, as we need to solve a different minimization problem for every \lambda . But for some classes of functions, it is possible to get an explicit formula for g(\lambda) . Solving the primal and dual programs together is often easier than solving only one of them. Examples are linear programming and quadratic programming. A better and more general approach to duality is provided by Fenchel's duality theorem. Another condition in which the min-max and max-min are equal is when the Lagrangian has a saddle point: (x^*, \lambda^*) is a saddle point of the Lagrange function L if and only if x^* is an optimal solution to the primal, \lambda^* is an optimal solution to the dual, and the optimal values in the indicated problems are equal to each other. &h_i(x) = 0,\ i \in \left \{1,\ldots,p \right \} \end{align} with the domain \mathcal{D} \subset \mathbb{R}^n having non-empty interior, the Lagrangian function \mathcal{L}: \mathbb{R}^n \times \mathbb{R}^m \times \mathbb{R}^p \to \mathbb{R} is defined as \mathcal{L}(x,\lambda,\nu) = f_0(x) + \sum_{i=1}^m \lambda_i f_i(x) + \sum_{i=1}^p \nu_i h_i(x). The vectors \lambda and \nu are called the dual variables or Lagrange multiplier vectors associated with the problem. The Lagrange dual function g:\mathbb{R}^m \times \mathbb{R}^p \to \mathbb{R} is defined as g(\lambda,\nu) = \inf_{x\in\mathcal{D}} \mathcal{L}(x,\lambda,\nu) = \inf_{x\in\mathcal{D}} \left\{ f_0(x) + \sum_{i=1}^m \lambda_i f_i(x) + \sum_{i=1}^p \nu_i h_i(x) \right\}. The dual function g is concave, even when the initial problem is not convex, because it is a point-wise infimum of affine functions. The dual function yields lower bounds on the optimal value p^* of the initial problem; for any \lambda \geq 0 and any \nu we have g(\lambda,\nu) \leq p^* . If a constraint qualification such as Slater's condition holds and the original problem is convex, then we have strong duality, i.e. d^* = \max_{\lambda \ge 0, \nu} g(\lambda,\nu) = \inf f_0 = p^* . & &g_i(x) \leq 0, \quad i = 1,\ldots,m \end{align} the Lagrangian dual problem is \begin{align} &\underset{u}{\operatorname{maximize}}& & \inf_x \left(f(x) + \sum_{j=1}^m u_j g_j(x)\right) \\ &\operatorname{subject\;to} & &u_i \geq 0, \quad i = 1,\ldots,m \end{align} where the objective function is the Lagrange dual function. Provided that the functions f and g_1, \ldots, g_m are continuously differentiable, the infimum occurs where the gradient is equal to zero. The problem \begin{align} &\underset{x, u}{\operatorname{maximize}}& & f(x) + \sum_{j=1}^m u_j g_j(x) \\ &\operatorname{subject\;to} & & \nabla f(x) + \sum_{j=1}^m u_j \, \nabla g_j(x) = 0 \\ &&&u_i \geq 0, \quad i = 1,\ldots,m \end{align} is called the Wolfe dual problem. This problem may be difficult to deal with computationally, because the objective function is not concave in the joint variables (u,x) . Also, the equality constraint \nabla f(x) + \sum_{j=1}^m u_j \, \nabla g_j(x) is nonlinear in general, so the Wolfe dual problem is typically a nonconvex optimization problem. In any case, weak duality holds.

See also

Convex duality Duality Relaxation (approximation)

Notes

Books

Articles

Duality in Linear Programming Gary D. Knott

Text derived from https://en.wikipedia.org/w/index.php?oldid=1306932479 under CC BY-SA 4.0; modifications have been made.

Provenance

snapshot enwiki-001306932479 · captured 1755693759.0 · version history

Cite this page as: encyclopegia.org, “Duality (optimization)”, snapshot enwiki-001306932479. https://encyclopegia.org/a/duality-optimization

Sources cited in this revision

6 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.