Lesson plan

binary entropy function

(in shannons) as a function of binary outcome probability, called the binary entropy function. In information theory, the binary entropy function, denoted \operatorname H(p) or \operatorname H_\text{b}(p) , is defined as the entropy of a Bernoulli process (i.i.d. binary variable) X with probability p of one of two values, and is given by the formula: : \operatorname H(X) = -p \log p - (1 - p) \log (1 - p). The base of the logarithm corresponds to the choice of units of information; base (natural logarithm) corresponds to nats and is mathematically convenient, while base 2 (binary logarithm) corresponds to shannons and is conventional (as shown in the graph); explicitly: : \operatorname H(X) = -p \log_2 p - (1 - p) \log_2 (1 - p). Note that the values at 0 and 1 are given by the limit \textstyle 0 \log 0 := \lim_{x \to 0^+} x \log x = 0 (by L'Hôpital's rule); and that "binary" refers to two possible values for the variable, not the units of information. When p = 1/2 , the binary entropy function attains its maximum value of 1 shannon (1 binary unit of information); this is the case of an unbiased coin flip. When p = 0 or p = 1 , the binary entropy is 0 (regardless of the unit), corresponding to no information, since there is no uncertainty in the variable.

Back to article

Article summary

(in shannons) as a function of binary outcome probability, called the binary entropy function. In information theory, the binary entropy function, denoted \operatorname H(p) or \operatorname H_\text{b}(p) , is defined as the entropy of a Bernoulli process (i.i.d. binary variable) X with probability p of one of two values, and is given by the formula: : \operatorname H(X) = -p \log p - (1 - p) \log (1 - p). The base of the logarithm corresponds to the choice of units of information; base (natural logarithm) corresponds to nats and is mathematically convenient, while base 2 (binary logarithm) corresponds to shannons and is conventional (as shown in the graph); explicitly: : \operatorname H(X) = -p \log_2 p - (1 - p) \log_2 (1 - p). Note that the values at 0 and 1 are given by the limit \textstyle 0 \log 0 := \lim_{x \to 0^+} x \log x = 0 (by L'Hôpital's rule); and that "binary" refers to two possible values for the variable, not the units of information. When p = 1/2 , the binary entropy function attains its maximum value of 1 shannon (1 binary unit of information); this is the case of an unbiased coin flip. When p = 0 or p = 1 , the binary entropy is 0 (regardless of the unit), corresponding to no information, since there is no uncertainty in the variable.

Key claims

No claims recorded.

Discussion questions

  1. Summarise the article in your own words. Which single claim would you verify first, and why?
  2. Which claim in this article is most likely to be revised as the sources update? How would you check its current status?
  3. If two sources disagreed about one of these facts, what would you need in order to trust one over the other?

Bibliography

No sources recorded.

Key claims and bibliography are generated from the same tracked claims and sources as the article page.