Lesson plan

Scoring rule

In decision theory, both a scoring rule as well as a scoring function provide an ex post summary measure for the evaluation of the quality of a prediction or forecast. They assign a numeric score to a single prediction given the actual outcome. Depending on the sign convention, this score can be interpreted as a loss or a reward for the forecaster. Scoring rules assess probabilistic predictions or forecasts, i.e. predictions of the whole probability distribution F of the outcome. On the other hand, scoring functions assess point predictions, i.e. predictions of a property or functional T(F) of the probability distribution F of the outcome. Examples of such a property are the expectation and the median. to the observed quantiles. Blue is the best calibrated model, see calibration (statistics). Scoring rules answer the question "how good is a predicted probability distribution given the observation of the actual outcome?" Scoring rules that are (strictly) proper are proven to have the lowest expected score if the predicted distribution equals the underlying distribution of the target variable. Although this might differ for individual observations, this should result in a minimization of the expected score if the "correct" distributions are predicted.…

Back to article

Article summary

In decision theory, both a scoring rule as well as a scoring function provide an ex post summary measure for the evaluation of the quality of a prediction or forecast. They assign a numeric score to a single prediction given the actual outcome. Depending on the sign convention, this score can be interpreted as a loss or a reward for the forecaster. Scoring rules assess probabilistic predictions or forecasts, i.e. predictions of the whole probability distribution F of the outcome. On the other hand, scoring functions assess point predictions, i.e. predictions of a property or functional T(F) of the probability distribution F of the outcome. Examples of such a property are the expectation and the median. to the observed quantiles. Blue is the best calibrated model, see calibration (statistics). Scoring rules answer the question "how good is a predicted probability distribution given the observation of the actual outcome?" Scoring rules that are (strictly) proper are proven to have the lowest expected score if the predicted distribution equals the underlying distribution of the target variable. Although this might differ for individual observations, this should result in a minimization of the expected score if the "correct" distributions are predicted.…

Key claims

No claims recorded.

Discussion questions

  1. Summarise the article in your own words. Which single claim would you verify first, and why?
  2. Which claim in this article is most likely to be revised as the sources update? How would you check its current status?
  3. If two sources disagreed about one of these facts, what would you need in order to trust one over the other?

Bibliography

No sources recorded.

Key claims and bibliography are generated from the same tracked claims and sources as the article page.