Smooth maximum
≈ 4 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
In mathematics, a smooth maximum of an indexed family x 1 , ..., x n of numbers is a smooth approximation to the maximum function \max(x_1,\ldots,x_n), meaning a parametric family of functions m_\alpha(x_1,\ldots,x_n) such that for every , the function is smooth, and the family converges to the maximum function as . The concept of smooth minimum is similarly defined. In many cases, a single family approximates both: maximum as the parameter goes to positive infinity, minimum as the parameter goes to negative infinity; in symbols, as and as . The term can also be used loosely for a specific smooth function that behaves similarly to a maximum, without necessarily being part of a parametrized family.
In mathematics, a smooth maximum of an indexed family x 1 , ..., x n of numbers is a smooth approximation to the maximum function \max(x_1,\ldots,x_n), meaning a parametric family of functions m_\alpha(x_1,\ldots,x_n) such that for every , the function is smooth, and the family converges to the maximum function as . The concept of smooth minimum is similarly defined. In many cases, a single family approximates both: maximum as the parameter goes to positive infinity, minimum as the parameter goes to negative infinity; in symbols, as and as . The term can also be used loosely for a specific smooth function that behaves similarly to a maximum, without necessarily being part of a parametrized family. \mathcal{S}_\alpha (x_1,\ldots,x_n) = \frac{\sum_{i=1}^n x_i e^{\alpha x_i}}{\sum_{i=1}^n e^{\alpha x_i}} \mathcal{S}_\alpha has the following properties: \mathcal{S}_\alpha\to \max as \alpha\to\infty \mathcal{S}_0 is the arithmetic mean of its inputs \mathcal{S}_\alpha\to \min as \alpha\to -\infty The gradient of \mathcal{S}_{\alpha} is closely related to softmax and is given by \nabla_{x_i}\mathcal{S}_\alpha (x_1,\ldots,x_n) = \frac{e^{\alpha x_i}}{\sum_{j=1}^n e^{\alpha x_j}} [1 + \alpha(x_i - \mathcal{S}_\alpha (x_1,\ldots,x_n))]. This makes the softmax function useful for optimization techniques that use gradient descent. This operator is sometimes called the Boltzmann operator, after the Boltzmann distribution. \mathrm{LSE}_\alpha(x_1, \ldots, x_n) = \frac{1}{\alpha} \log \sum_{i=1}^n \exp \alpha x_i This can also be normalized if the x_i are all non-negative, yielding a function with domain [0,\infty)^n and range [0, \infty) : g(x_1, \ldots, x_n) = \log \left( \sum_{i=1}^n \exp x_i - (n-1) \right) The (n - 1) term corrects for the fact that \exp(0) = 1 by canceling out all but one zero exponential, and \log 1 = 0 if all x_i are zero. \mathrm{mm}_\alpha(x) = \frac{1}{\alpha} \log \frac{1}{n} \sum_{i=1}^n \exp \alpha x_i It is a non-expansive operator. As \alpha \to \infty , it acts like a maximum. As \alpha \to 0 , it acts like an arithmetic mean. As \alpha \to -\infty , it acts like a minimum. This operator can be viewed as a particular instantiation of the quasi-arithmetic mean. It can also be derived from information theoretical principles as a way of regularizing policies with a cost function defined by KL divergence. The operator has previously been utilized in other areas, such as power engineering. \| (x_1, \ldots, x_n) \|_p = \left( \sum_{i=1}^n |x_i|^p \right)^\frac{1}{p} which converges to \| (x_1, \ldots, x_n) \|_\infty = \max_{1\leq i\leq n} |x_i| as p \to \infty . An advantage of the p-norm is that it is a norm. As such it is scale invariant (homogeneous): \| (\lambda x_1, \ldots, \lambda x_n) \|_p = |\lambda| \cdot \| (x_1, \ldots, x_n) \|_p , and it satisfies the triangle inequality. &= \frac{a + b + |a - b|_\varepsilon}{2} \\ &= \frac{a + b + \sqrt{(a - b)^2 + \varepsilon}}{2} \end{align} where \varepsilon \geq 0 is a parameter. As \varepsilon \to 0 , |\cdot|_\varepsilon \to |\cdot| and thus \textstyle\max_\varepsilon \to \max .
See also
LogSumExp Softmax function Generalized mean
References
https://www.johndcook.com/soft_maximum.pdf M. Lange, D. Zühlke, O. Holz, and T. Villmann, "Applications of lp-norms and their smooth approximations for gradient based learning vector quantization," in Proc. ESANN, Apr. 2014, pp. 271-276. (https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-153.pdf)
Text derived from https://en.wikipedia.org/w/index.php?oldid=1317993932 under CC BY-SA 4.0; modifications have been made.