Top-p sampling

Uses your system voice; voice and rate in settings.

Snapshot

≈ 3 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

Top-p sampling, also known as nucleus sampling, is a stochastic decoding strategy for generating sequences from autoregressive probabilistic models. It was originally proposed by Ari Holtzman, Yejin Choi and colleagues in 2019 for natural language generation to address the issue of repetitive and nonsensical text generated by other common decoding methods like beam search. The technique has since been applied in other scientific fields, such as protein engineering and geophysics.

How it works

In top-p sampling, a probability threshold p is set, and the next item in a sequence is sampled only from the smallest possible set of high-probability candidates whose cumulative probability exceeds p. This method adapts the size of the candidate pool based on the model's certainty, making it more flexible than top-k sampling, which samples from a fixed number of candidates. Due to its effectiveness, top-p sampling is a widely used technique in many large language model applications.

Top-p sampling, also known as nucleus sampling, is a stochastic decoding strategy for generating sequences from autoregressive probabilistic models. It was originally proposed by Ari Holtzman, Yejin Choi and colleagues in 2019 for natural language generation to address the issue of repetitive and nonsensical text generated by other common decoding methods like beam search. The technique has since been applied in other scientific fields, such as protein engineering and geophysics.

In top-p sampling, a probability threshold p is set, and the next item in a sequence is sampled only from the smallest possible set of high-probability candidates whose cumulative probability exceeds p. This method adapts the size of the candidate pool based on the model's certainty, making it more flexible than top-k sampling, which samples from a fixed number of candidates.

Due to its effectiveness, top-p sampling is a widely used technique in many large language model applications. \sum_{x \in V^{(p)}} P(x|x_{1}, \dots, x_{t-1}) \geq p In this formula, P(x|x_{1}, \dots, x_{t-1}) represents the probability of a token x given the preceding tokens x_{1}, \dots, x_{t-1} . If we set p = 0.8 : The tokens are sorted by probability: [the, a, cat, dog, eats].

The cumulative probability is calculated: the: 0.5 the + a: 0.5 + 0.2 = 0.7 the + a + cat: 0.7 + 0.1 = 0.8 The nucleus is the smallest set with cumulative probability ≥ 0.8, which is V^{(0.8)} = \{\text{the, a, cat}\} . The probabilities for this set are rescaled to sum to 1: P(the) = 0.5 / 0.8 = 0.625 P(a) = 0.2 / 0.8 = 0.25 P(cat) = 0.1 / 0.8 = 0.125 The next token is then sampled from this new distribution, meaning dog and eats have a 0% chance of being chosen.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1368853719 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

5 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.