Smooth maximum

In mathematics, a smooth maximum of an indexed family x 1 , ..., x n of numbers is a smooth approximation to the maximum function \max(x_1,\ldots,x_n), meaning a parametric family of functions m_\alpha(x_1,\ldots,x_n) such that for every…

Uses your system voice; voice and rate in settings.

≈ 5 min read

Cite, export and learn8 citation formats · 8 exports · lesson and quiz

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

In mathematics, a smooth maximum of an indexed family x1, ..., xn of numbers is a smooth approximation to the maximum function max⁡(x1,…,xn), meaning a parametric family of functions mα(x1,…,xn) such that for every ⁠α⁠, the function ⁠mα⁠ is smooth, and the family converges to the maximum function ⁠mα→max⁠ as ⁠α→∞⁠. The concept of smooth minimum is similarly defined. In many cases, a single family approximates both: maximum as the parameter goes to positive infinity, minimum as the parameter goes to negative infinity; in symbols, ⁠mα→max⁠ as ⁠α→∞⁠ and ⁠mα→min⁠ as ⁠α→−∞⁠. The term can also be used loosely for a specific smooth function that behaves similarly to a maximum, without necessarily being part of a parametrized family.

Examples

edit edit source

Boltzmann operator

edit edit source

For large positive values of the parameter α>0, the following formulation is a smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

𝒮α(x1,…,xn)=∑i=1nxieαxi∑i=1neαxi

𝒮α has the following properties:

  1. 𝒮α→max as α→∞
  2. 𝒮0 is the arithmetic mean of its inputs
  3. 𝒮α→min as α→−∞

The gradient of 𝒮α is closely related to softmax and is given by

∇xi𝒮α(x1,…,xn)=eαxi∑j=1neαxj[1+α(xi−𝒮α(x1,…,xn))].

This makes the softmax function useful for optimization techniques that use gradient descent.

This operator is sometimes called the Boltzmann operator,[1] after the Boltzmann distribution.

LogSumExp

edit edit source

Another smooth maximum is LogSumExp:

LSEα(x1,…,xn)=1αlog⁡∑i=1nexp⁡αxi

This can also be normalized if the xi are all non-negative, yielding a function with domain [0,∞)n and range [0,∞):

g(x1,…,xn)=log⁡(∑i=1nexp⁡xi−(n−1))

The (n−1) term corrects for the fact that exp⁡(0)=1 by canceling out all but one zero exponential, and log⁡1=0 if all xi are zero.

Mellowmax

edit edit source

The mellowmax operator[1] is defined as follows:

mmα(x)=1αlog⁡1n∑i=1nexp⁡αxi

It is a non-expansive operator. As α→∞, it acts like a maximum. As α→0, it acts like an arithmetic mean. As α→−∞, it acts like a minimum. This operator can be viewed as a particular instantiation of the quasi-arithmetic mean. It can also be derived from information theoretical principles as a way of regularizing policies with a cost function defined by KL divergence. The operator has previously been utilized in other areas, such as power engineering.[2]

Connection between LogSumExp and Mellowmax

edit edit source

LogSumExp and Mellowmax are the same function differing by a constant log⁡nα. LogSumExp is always larger than the true max, differing at most from the true max by log⁡nα in the case where all n arguments are equal and being exactly equal to the true max when all but one argument is −∞. Similarly, Mellowmax is always less than the true max, differing at most from the true max by log⁡nα in the case where all but one argument is −∞ and being exactly equal to the true max when all n arguments are equal.

p-Norm

edit edit source

Another smooth maximum is the p-norm:

‖(x1,…,xn)‖p=(∑i=1n|xi|p)1p

which converges to ‖(x1,…,xn)‖∞=max1≤i≤n|xi| as p→∞.

An advantage of the p-norm is that it is a norm. As such it is scale invariant (homogeneous): ‖(λx1,…,λxn)‖p=|λ|⋅‖(x1,…,xn)‖p, and it satisfies the triangle inequality.

Smooth maximum unit

edit edit source

The following binary operator is called the Smooth Maximum Unit (SMU):[3]

maxε(a,b)=a+b+|a−b|ε2=a+b+(a−b)2+ε2

where ε≥0 is a parameter. As ε→0, |⋅|ε→|⋅| and thus maxε→max.

See also

edit edit source

References

edit edit source
  1. ^ a b Asadi, Kavosh; Littman, Michael L. (2017). "An Alternative Softmax Operator for Reinforcement Learning". PMLR. 70: 243–252. arXiv:1612.05628. Retrieved January 6, 2023. Wikipedia revision 1317993932, retrieved 2026-11-18
  2. ^ Safak, Aysel (February 1993). "Statistical analysis of the power sum of multiple correlated log-normal components". IEEE Transactions on Vehicular Technology. 42 (1): {58–61. doi:10.1109/25.192387. Wikipedia revision 1317993932, retrieved 2026-11-18
  3. ^ Biswas, Koushik; Kumar, Sandeep; Banerjee, Shilpak; Ashish Kumar Pandey (2021). "SMU: Smooth activation function for deep networks using smoothing maximum technique". arXiv:2111.04682 [cs.LG]. Wikipedia revision 1317993932, retrieved 2026-11-18

https://www.johndcook.com/soft_maximum.pdf

M. Lange, D. Zühlke, O. Holz, and T. Villmann, "Applications of lp-norms and their smooth approximations for gradient based learning vector quantization," in Proc. ESANN, Apr. 2014, pp. 271-276. (https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-153.pdf)

Text derived from https://en.wikipedia.org/w/index.php?oldid=1317993932 under CC BY-SA 4.0; modifications have been made.

Words in each sectionExamplesExamplesExamples: 515 words515See alsoSee alsoSee also: 8 words8
Words in each section. Measured from the stored article body; sections below the minimum are not plotted.

Provenance

snapshot enwiki-001317993932 · captured 2025-10-21 · ledger · version history

Cite this page as: encyclopegia.org, “Smooth maximum”, snapshot enwiki-001317993932. https://encyclopegia.org/a/smooth-maximum