二元熵函数

Uses your system voice; voice and rate in settings.

The name shown is in zh. The article below is in English and is not a translation.

Q4913893Snapshot

≈ 5 min read

Names in other languages

  • FrançaisFonction d'entropie binaire
  • caFunció d'entropia binària
  • Українськафункція бінарної ентропіїфункція двійкової ентропії
  • Englishbinary entropy function
  • PolskiEntropia binarna
  • 日本語二進エントロピー函数
  • 中文二元熵函数
  • 简体中文二元熵函数
  • 繁體中文二元熵函式

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

(Learn how and when to remove this message)

In information theory, the binary entropy function, denoted H⁡(p) or Hb(p), is defined as the entropy of a Bernoulli process (i.i.d. binary variable) X with probability p of one of two values, and is given by the formula:

H⁡(X)=−plog⁡p−(1−p)log⁡(1−p).

The base of the logarithm corresponds to the choice of units of information; base e (natural logarithm) corresponds to nats and is mathematically convenient, while base 2 (binary logarithm) corresponds to shannons and is conventional (as shown in the graph); explicitly:

H⁡(X)=−plog2p−(1−p)log2(1−p).

Note that the values at 0 and 1 are given by the limit 0log⁡0:=limx→0+xlog⁡x=0 (by L'Hôpital's rule); and that "binary" refers to two possible values for the variable, not the units of information.

When p=1/2, the binary entropy function attains its maximum value of 1 shannon (1 binary unit of information); this is the case of an unbiased coin flip. When p=0 or p=1, the binary entropy is 0 (regardless of the unit), corresponding to no information, since there is no uncertainty in the variable.

Notation

edit edit source

Binary entropy Hb(p) is a special case of H(X), the entropy function. Hb(p) is distinguished from the general entropy function H(X) in that the former takes a single real number as a parameter whereas the latter takes a distribution or random variable as a parameter. Thus the binary entropy (of p) is the entropy of the particular distribution X∼Ber⁡(p), so Hb(p)=H(Ber⁡(p)).

Writing the probability of each of the two values being p and q, so p+q=1 and q=1−p, this corresponds to

H⁡(X)=−plog⁡p−(1−p)log⁡(1−p)=−plog⁡p−qlog⁡q=−∑x∈XPr⁡(X=x)⋅log⁡Pr⁡(X=x)=H(Ber⁡(p)).

Sometimes the binary entropy function is also written as H2(p). However, it is different from and should not be confused with the Rényi entropy, which is also denoted as H2(X).

Explanation

edit edit source

In terms of information theory, entropy is considered to be a measure of the uncertainty in a message. To put it intuitively, suppose p=0. At this probability, the event is certain never to occur, and so there is no uncertainty at all, leading to an entropy of 0. If p=1, the result is again certain, so the entropy is 0 here as well. When p=1/2, the uncertainty is at a maximum; if one were to place a fair bet on the outcome in this case, there is no advantage to be gained with prior knowledge of the probabilities. In this case, the entropy is maximum at a value of 1 bit. Intermediate values fall between these cases; for instance, if p=1/4, there is still a measure of uncertainty on the outcome, but one can still predict the outcome correctly more often than not, so the uncertainty measure, or entropy, is less than 1 full bit.

Properties

edit edit source

Derivative

edit edit source

The derivative of the binary entropy function may be expressed as the negative of the logit function:

ddpHb(p)=−logita(p)=−loga(p1−p).
d2dp2Hb(p)=−1p(1−p)ln⁡a,

where a denotes the given base of the logarithm.

Convex conjugate

edit edit source

The convex conjugate (specifically, the Legendre transform) of the binary entropy (with base e) is the negative softplus function. This is because (following the definition of the Legendre transform: the derivatives are inverse functions) the derivative of negative binary entropy is the logit, whose inverse function is the logistic function, which is the derivative of softplus.

Softplus can be interpreted as logistic loss, so by duality, minimizing logistic loss corresponds to maximizing entropy. This justifies the principle of maximum entropy as loss minimization.

Taylor series

edit edit source

The Taylor series of the binary entropy function at 1/2 is

Hb(p)=1−12ln⁡2∑n=1∞(1−2p)2nn(2n−1)

which converges to the binary entropy function for all values 0≤p≤1.

Bounds

edit edit source

The following bounds hold for 0<p<1:[1]

ln⁡(2)⋅log2(p)⋅log2(1−p)≤Hb(p)≤log2(p)⋅log2(1−p)

and

4p(1−p)≤Hb(p)≤(4p(1−p))(1/ln⁡4)

where ln denotes natural logarithm.

See also

edit edit source

References

edit edit source
  1. ^ Topsøe, Flemming (2001). "Bounds for entropy and divergence for distributions over a two-element set". JIPAM. Journal of Inequalities in Pure & Applied Mathematics. 2 (2): Paper No. 25, 13 p.-Paper No. 25, 13 p.

Further reading

edit edit source

Text derived from https://en.wikipedia.org/w/index.php?oldid=1324263438 under CC BY-SA 4.0; modifications have been made.

Provenance

snapshot binary-entropy-function-560930-33e68af1cbe5dc83 · hash 33e68af1cbe5dc83e8462ef3c90265c32fbef2db7079dfb09896f928d621aa0f · captured 1764169661.0 · version history

Cite this page as: encyclopegia.org, “二元熵函数”, snapshot binary-entropy-function-560930-33e68af1cbe5dc83. https://encyclopegia.org/a/binary-entropy-function