Scoring rule
≈ 17 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
In decision theory, both a scoring rule as well as a scoring function provide an ex post summary measure for the evaluation of the quality of a prediction or forecast. They assign a numeric score to a single prediction given the actual outcome. Depending on the sign convention, this score can be interpreted as a loss or a reward for the forecaster. Scoring rules assess probabilistic predictions or forecasts, i.e. predictions of the whole probability distribution F of the outcome. On the other hand, scoring functions assess point predictions, i.e. predictions of a property or functional T(F) of the probability distribution F of the outcome. Examples of such a property are the expectation and the median. to the observed quantiles. Blue is the best calibrated model, see calibration (statistics). Scoring rules answer the question "how good is a predicted probability distribution given the observation of the actual outcome?" Scoring rules that are (strictly) proper are proven to have the lowest expected score if the predicted distribution equals the underlying distribution of the target variable. Although this might differ for individual observations, this should result in a minimization of the expected score if the "correct" distributions are predicted.…
In decision theory, both a scoring rule as well as a scoring function provide an ex post summary measure for the evaluation of the quality of a prediction or forecast. They assign a numeric score to a single prediction given the actual outcome. Depending on the sign convention, this score can be interpreted as a loss or a reward for the forecaster. Scoring rules assess probabilistic predictions or forecasts, i.e. predictions of the whole probability distribution F of the outcome. On the other hand, scoring functions assess point predictions, i.e. predictions of a property or functional T(F) of the probability distribution F of the outcome. Examples of such a property are the expectation and the median. to the observed quantiles. Blue is the best calibrated model, see calibration (statistics).]]Scoring rules answer the question "how good is a predicted probability distribution given the observation of the actual outcome?" Scoring rules that are (strictly) proper are proven to have the lowest expected score if the predicted distribution equals the underlying distribution of the target variable. Although this might differ for individual observations, this should result in a minimization of the expected score if the "correct" distributions are predicted. In the same way, scoring functions answer the question "how good is a point prediction given the observation of the actual outcome?". Scoring functions that are (strictly) consistent (for the functional T ) are proven to have the lowest expected score if the point prediction equals (or is among) the true functional of the underlying distribution of the target variable. Scoring rules and scoring functions are often used as "cost functions" or "loss functions" of forecasting models. If a sample of forecasts and observations of the outcome is collected, they can be evaluated as the empirical mean of the given sample, often also called the "score". Scores of predictions of different models or forecasters can then be compared to conclude which model or forecaster is best. For example, consider a probabilistic model that predicts (based on an input x ) a gaussian distribution \mathcal{N}(\mu, \sigma^2) with mean \mu \in \mathbb{R} and standard deviation \sigma \in \mathbb{R}_+ . A common interpretation of probabilistic models is that they aim to quantify their own predictive uncertainty. In this example, an observed target variable y \in \mathbb{R} is then held compared to the predicted distribution \mathcal{N}(\mu, \sigma^2) and assigned a score \mathbf{S}(\mathcal{N}(\mu, \sigma^2), y) \in \mathbb{R} . When a probabilistic model is trained on a scoring rule, it should "teach" the model to predict when its uncertainty is low, and when its uncertainty is high, and it should result in calibrated predictions, while minimizing the predictive uncertainty. Although the example given concerns the probabilistic forecasting of a real valued target variable, a variety of different scoring rules have been designed with different target variables in mind. Scoring rules exist for binary and categorical probabilistic classification, as well as for univariate and multivariate probabilistic regression. Consider a sample space or observation domain, \Omega , which comprises the potential outcomes of a future observation; a σ-algebra \mathcal A of subsets of \Omega and a convex class \mathcal F of probability measures on (\Omega, \mathcal A) . A function defined on \Omega and taking values in the extended real line, \overline{\mathbb{R}} = [-\infty, \infty] , is \mathcal F -quasi-integrable if it is measurable with respect to \mathcal A and is quasi-integrable with respect to all F \in \mathcal{F} . A (statistical) functional T is a potentially set-valued mapping from the class of probability distributions \mathcal F to a Euclidean space, i.e. T\colon \mathcal F \rightarrow \mathbb{R}^d with F \rightarrow T(F) . \mathbb{E}_{Y \sim Q}[\mathbf{S}(F,Y)]= \int \mathbf{S}(F, \omega) \mathrm{d}Q(\omega) Similar, the expected score of a point prediction x \in \mathbb{R}^d with resprect to the underlying distribution Q \in \mathcal{F} : \mathbb{E}_{Y \sim Q}[S(x,Y)]= \int S(x, \omega) \mathrm{d}Q(\omega) A way to estimate the expected score is by means of the sample average score. Given a sample of prediction-observation pairs e.g. (F_i, y_i) for probabilistic predictions F_i and observations y_i \in \Omega , i=1\ldots n , (x_i, y_i) for point predictions x_i , the average score is calculated as for scoring rules: \widehat{E[\mathbf{S}]} = \frac{1}{n}\sum_{i=1}^n \mathbf{S}(F_i, y_i) for scoring functions: \widehat{E[S]} = \frac{1}{n}\sum_{i=1}^n S(x_i, y_i) By invoking some law of large numbers argument, the sample average scores are consistent estimators of the expectation. Strictly proper scoring rules and strictly consistent scoring functions encourage honest forecasts by maximization of the expected reward: If a forecaster is given a reward of -\mathbf{S}(F, y) if y realizes (e.g. y=rain ), then the highest expected reward (lowest score) is obtained by reporting the true probability distribution.
Proper scoring rules
A scoring rule \mathbf{S} is proper relative to \mathcal{F} if (assuming negative orientation) its expected score is minimized when the forecasted distribution matches the distribution of the observation. \mathbb{E}_{Y \sim Q}[\mathbf{S}(Q, Y)] \leq \mathbb{E}_{Y \sim Q}[\mathbf{S}(F, Y)] for all F,Q \in\mathcal{F} . It is strictly proper if the above equation holds with equality if and only if F=Q . E(S) = \mathrm{UNC} + \mathrm{REL} - \mathrm{RES}. If a score is proper and negatively oriented (such as the Brier Score), all three terms are positive definite. The uncertainty component is equal to the expected score of the forecast which constantly predicts the average event frequency. The reliability component penalizes poorly calibrated forecasts, in which the predicted probabilities do not coincide with the event frequencies. The equations for the individual components depend on the particular scoring rule. For the Brier Score, they are given by \mathrm{UNC} = \bar{x}(1-\bar{x}) \mathrm{REL} = E(p-\pi(p))^2 \mathrm{RES} = E(\pi(p)-\bar{x})^2 where \bar{x} is the average probability of occurrence of the binary event x , and \pi(p) is the conditional event probability, given p , i.e. \pi(p) = P(x=1\mid p) For a categorical response variable with m mutually exclusive events, Y \in \Omega = \{1, \ldots, m\} , a probabilistic forecaster or algorithm will return a probability vector \mathbf{p} \in [0,1]^m with probabilities for each of the m outcomes. If y=i materializes, one often abbreviates the score as \mathbf{S}(\mathbf{p}, i) . \mathbf{S}(\mathbf{p}, i) = \ln(p_i) Here, the score is calculated as the logarithm of the probability estimate for the actual outcome. That is, a prediction of 80% that correctly proved true would receive a score of . This same prediction also assigns 20% likelihood to the opposite case, and so if the prediction proves false, it would receive a score based on the 20%: . The goal of a forecaster is to maximize the score and for the score to be as large as possible, and −0.22 is indeed larger than −1.6. If one treats the truth or falsity of the prediction as a variable with value 1 or 0 respectively, and the expressed probability as , then one can write the logarithmic scoring rule as . Note that any logarithmic base may be used, since strictly proper scoring rules remain strictly proper under linear transformation. That is: L(\mathbf{p},i) = \log_b(p_i) is strictly proper for all b>1 . \mathbf{S}_Q(\mathbf{p},i) = 2p_i - \mathbf{p}\cdot \mathbf{p} = 2p_i -\sum_{j=1}^m p_j^2 where p_i is the probability assigned to the correct answer i . The Brier score, originally proposed by Glenn W. Brier in 1950, can be obtained by an affine transform from the quadratic scoring rule. \mathbf{S}_B(\mathbf{p},i) = \sum_{j=1}^m (y_j-p_j)^2 Where y_j = 1 when the j th event is correct and y_j = 0 otherwise. It can be thought of as a generalization of mean squared error to probabilistic forecasts. An important difference between these two rules is that a forecaster should strive to maximize the quadratic score \mathbf{S}_Q yet minimize the Brier score \mathbf{S}_B . This is due to a negative sign in the linear transformation between them. \mathbf{S}(\mathbf{p},i) = \frac{p_i}{\lVert \mathbf{p} \rVert} = \frac{p_i}{\sqrt{p_1^2 + \cdots + p_m^2}} Also its generalization with \alpha > 1 is strictly proper \mathbf{S}(\mathbf{p},i) = \frac{p_i^{\alpha-1}}{\left(\sum_{j=1}^m p_j^\alpha\right)^{(\alpha-1)/\alpha}} RPS(\mathbf{p},i) = \sum_{k=1}^{m-1} \left(\sum_{j=1}^k p_j - y_j\right)^2 Where y_j = 1 when the j th event is correct and y_j = 0 otherwise, and m is the number of classes. Other than other scoring rules, the ranked probability score considers the distance between classes, i.e. classes 1 and 2 are considered closer than classes 1 and 3. The score assigns better scores to probabilistic forecasts with high probabilities assigned to classes close to the correct class. For example, when considering probabilistic forecasts \mathbf{p}_1 = (0.5, 0.5, 0) and \mathbf{p}_2 = (0.5, 0, 0.5) , we find that RPS(\mathbf{p}_1,1) = 0.25 , while RPS(\mathbf{p}_2,1) = 0.5 , despite both probabilistic forecasts assigning identical probability to the correct class. L(F,y) = - \ln(f(y)) . The logarithmic score for continuous variables has strong ties to Maximum likelihood estimation and to the Kullback–Leibler divergence. S(f,y)= 2 f(y) - \|f\|_2^2 It is strictly proper for densities for which the norm \|f\|_2 = \left(\int f(y)^2 dy\right)^{\frac{1}{2}} exists. CRPS(F,y)=\int_\mathbb{R} ( F(x) - H(x - y) ) ^2 dx where H is the Heaviside step function and y \in \mathbb R is the observation. For distributions with finite first moment, the continuous ranked probability score can be written as: CRPS(F, y) = \mathbb{E}_{X \sim F}|X-y| - \frac{1}{2}\mathbb{E}_{X,X' \sim F}|X-X'| where X and X' are independent random variables, both sampled from the distribution F . This is the energy form of CRPS and opens the door to estimating the CRPS via Monte Carlo sampling (through approximating the expectation value). Furthermore, when the cumulative probability function F is continuous, the continuous ranked probability score can also be written as CRPS(F, y) = \mathbb{E}_{X \sim F}|X-y| + \mathbb{E}_{X \sim F}[X] - 2 \mathbb{E}_{X \sim F}[X \cdot F(X)] The continuous ranked probability score can be seen as both a continuous extension of the ranked probability score, as well as quantile regression. The continuous ranked probability score over the empirical distribution \hat F_q of an ordered set points q_1 \leq \ldots \leq q_n (i.e. every point has 1/n probability of occurring), is equal to twice the mean quantile loss applied on those points with evenly spread quantiles (\tau_1, \ldots, \tau_n) = (1/(2n), \ldots, (2n-1)/(2n)) : CRPS\left(\hat F_q, y\right) = \frac{2}{n} \sum_{i=1}^n \tau_i (y - q_i)_+ + (1 - \tau_i) (q_i - y)_+ For many popular families of distributions, closed-form expressions for the continuous ranked probability score have been derived. The continuous ranked probability score has been used as a loss function for artificial neural networks, in which weather forecasts are postprocessed to a Gaussian probability distribution. CRPS was also adapted to survival analysis to cover censored events. The CRPS can be thought of as the generalization of the mean absolute error (MAE) to probabilistic forecasts, and for a single sample is equivalent to the MAE. Another way to think of it is the Brier/quadaratic score of the sampled cumulative distribution function F for the binary event \{X \leq y\} . CRPS is a special case of the Cramér distance and can be seen as an improvement of Wasserstein distance often used in machine learning. Cramér distance performed better in ordinal regression than KL distance or the Wasserstein metric. CRPS is widely used for evaluating probabilistic forecasts and compared against other scoring rules, see for example . It also has some critical theoretical limitations: It has been shown that CRPS can produce systematically misleading evaluations by favoring probabilistic forecasts whose medians are close to the observed outcome, regardless of the actual probability assigned to that region, potentially resulting in higher scores for forecasts that allocate negligible (or even zero) probability mass to the true outcome. Furthermore, CRPS is not invariant under smooth transformations of the forecast variable, and its ranking of forecast systems may reverse under such transformations, raising concerns about its consistency for evaluation purposes. S(F, y) = \left(\frac{y-\mu_F}{\sigma_F}\right)^2 + \log\sigma_F^2 L(D,y) = - \ln(f_D(y)) where f_D denotes the probability density function of the predicted multivariate distribution D . It is a local, strictly proper scoring rule. s(p) = 2 \Delta_y \log p(y) + \|\nabla_y \log p(y)\|_2^2 Where \Delta denotes the Hessian trace and \nabla denotes the gradient. This scoring rule can be used to computationally simplify parameter inference and address Bayesian model comparison with arbitrarily-vague priors. It was also used to introduce new information-theoretic quantities beyond the existing information theory. The Hyvärinen scoring rule is local of order 2 (meaning it locally takes into account derivatives up to second order). ES_\beta(D, Y) = \mathbb{E}_{X \sim D}[\lVert X - Y \rVert_2^\beta] - \frac{1}{2} \mathbb{E}_{X,X' \sim D}[\lVert X - X' \rVert_2^\beta] Here, \beta \in (0, 2) , \lVert\rVert_2 denotes the n -dimensional Euclidean distance and X, X' are independently sampled random variables from the probability distribution D . The energy score is strictly proper for distributions D for which \mathbb{E}_{X \sim D}[\lVert X \rVert_2] is finite. It has been suggested that the energy score is somewhat ineffective when evaluating the intervariable dependency structure of the forecasted multivariate distribution. Apart from a term that depends only on the distribution of the observation, the energy score is equal to twice the energy distance between the predicted distribution and the empirical distribution of the observation. VS_p(D, Y) = \sum_{i,j=1}^n w_{ij} (|Y_i - Y_j|^p - \mathbb{E}_{X \sim D}[|X_i - X_j|^p])^2 Here, w_{ij} are weights, often set to 1, and p > 0 can be arbitrarily chosen, but p = 0.5, 1 or 2 are often used. X_{i} is here to denote the i 'th marginal random variable of X . The variogram score is proper for distributions for which the (2p) 'th moment is finite for all components, but is never strictly proper. Compared to the energy score, the variogram score is claimed to be more discriminative with respect to the predicted correlation structure. CCRPS_{\mathcal{T}}(D,Y) = \sum_{i=1}^k CRPS(P_{X \sim D}(X_{v_i} | X_j = Y_j \text{ for } j \in \mathcal{C}_i), Y_{v_i}) Here, X_i is the i 'th marginal variable of X \sim D , \mathcal{T} = (v_i, \mathcal{C}_i)_{i=1}^k is a set of tuples that defines a conditional specification (with v_i \in \{1, \ldots, n\} and \mathcal{C}_i \subseteq \{1, \ldots, n\} \setminus \{v_i\} ), and P_{X \sim D}(X_{v_i} | X_j = Y_j \text{ for } j \in \mathcal{C}_i) denotes the conditional probability distribution for X_{v_i} given that all variables X_j for j \in \mathcal{C}_i are equal to their respective observations. In the case that P_{X \sim D}(X_{v_i} | X_j = Y_j \text{ for } j \in \mathcal{C}_i) is ill-defined (i.e. its conditional event has zero likelihood), CRPS scores over this distribution are defined as infinite. Conditional CRPS is strictly proper for distributions with finite first moment, if the chain rule is included in the conditional specification, meaning that there exists a permutation \phi_1, \ldots, \phi_n of 1, \ldots, n such that for all 1 \leq i \leq n : (\phi_i, \{\phi_1, \ldots, \phi_{i-1}\}) \in \mathcal{T} . The following scoring functions are strictly consistent for the expected value, i.e. T(F) = E_{Y\sim F}[Y] . S(x, y) = (x - y)^2 The following scoring functions are strictly consistent for the \alpha -quantile, i.e. T(F)=q_\alpha is the set of values q satisfying \lim_{y\uparrow q} F(y) \leq \alpha \leq F(q) . S(x, y) = (\mathbf{1}\{x \geq y\} - \alpha) (x - y) The point prediction consists of a central (1-\alpha) prediction interval, x=(l, u) , where the lower endpoint l predicts the \frac{\alpha}{2} quantile and the upper endpoint u predicts the 1-\frac{\alpha}{2} quantile. = (u - l) + \frac{2}{\alpha}(l - y)\,\mathbf{1}\{y u\} "The forecaster is rewarded for narrow prediction intervals, and he or she incurs a penalty, the size of which depends on α, if the observation misses the interval" Strictly Proper Scoring Rules, Prediction, and Estimation. Tilmann Gneiting &Adrian E Raftery Pages 359-378, https://doi.org/10.1198/016214506000001437, pdf Scoring rules for continuous probability distributions, James E. Matheson, Robert L. Winkler, (1976) Scoring Rules for Continuous Probability Distributions. Management Science 22(10):1087-1096. http://dx.doi.org/10.1287/mnsc.22.10.1087
External links
Video comparing spherical, quadratic and logarithmic scoring rules Local Proper Scoring Rules Scoring Rules and Decision Analysis Education Strictly Proper Scoring Rules Scoring Rules and uncertainty Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules Closed-form expressions of the continuous ranked probability score
Text derived from https://en.wikipedia.org/w/index.php?oldid=1375919887 under CC BY-SA 4.0; modifications have been made.