Reasoning model
≈ 2 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
Reasoning language models (RLMs) or large reasoning models (LRMs) are large language models that are trained to solve tasks that require several steps of reasoning. They tend to do better on logic, math, and programming tasks than standard LLMs, can revisit and revise earlier steps, make use of extra computations, and enhance performance when increasing the number of parameters, training examples, and training computations.
Reasoning language models (RLMs) or large reasoning models (LRMs) are large language models that are trained to solve tasks that require several steps of reasoning. They tend to do better on logic, math, and programming tasks than standard LLMs, can revisit and revise earlier steps, make use of extra computations, and enhance performance when increasing the number of parameters, training examples, and training computations. An outcome reward model, or outcome-supervised RM (ORM), gives the reward for a step r(x, y_1, \dots, y_i) based on the final answer: r(x, y_1, \dots, y_i) = r(x, y_n) . Such models are often called "verifiers". For tasks with answers that are easy to verify, such as math word problems, the outcome reward can be binary: 1 if the final answer is correct, 0 otherwise. If automatic verification is hard, humans can label answers as correct or incorrect, and those labels can be used to fine-tune a base model that predicts the human label. For tasks such as creative writing, where quality is not simply true or false, one can train a reward model on human ranked preference data, as in reinforcement learning from human feedback. A base model can also be fine-tuned to predict, from a partial thinking trace x, y_1, \dots, y_m , whether the final answer will be correct, and this prediction can serve as a binary reward. The ORM is usually trained with logistic regression, i.e. by minimizing cross-entropy loss. Given a process reward model (PRM} (see following), an ORM can be constructed by multiplying the total process reward during the reasoning trace by taking the minimum or by other ways of aggregating process rewards. DeepSeek used a simple ORM to train the R1 model.
Text derived from https://en.wikipedia.org/w/index.php?oldid=1377742798 under CC BY-SA 4.0; modifications have been made.