Gneiting-Raftery (2007) Strictly Proper Scoring Rules, Prediction, and Estimation

Summary

This JASA review article develops the theory of proper scoring rules — numerical rewards $S(P,x)$ that grade a probabilistic forecast $P$ against the outcome $x$ that materializes. A rule is proper if the forecaster maximizes expected score by quoting his or her true belief, and strictly proper if that optimum is unique; propriety is what makes a score a trustworthy tool for both evaluation (ranking competing forecasters/models) and estimation. The paper gives a general characterization of proper scoring rules via convex functions and Bregman divergences, catalogues the standard rules for categorical, density, quantile, and interval forecasts (logarithmic, Brier/quadratic, spherical, CRPS, energy score, interval score), and draws bridges to Bayes factors, cross-validation, and M-estimation. A weather-forecasting case study shows how an improper score can rank a deliberately dishonest forecast above an honest one.

Key Claims

• Definition of propriety. With $S(P,Q)=\int S(P,\omega)\,dQ(\omega)$ the expected score under $Q$ when $P$ is quoted, $S$ is proper relative to a class $\mathcal P$ if $S(Q,Q)\ge S(P,Q)$ for all $P,Q\in\mathcal P$, and strictly proper if equality holds iff $P=Q$. This is the incentive-compatibility property: honesty is optimal.
• Characterization theorem (Thm 1). A regular scoring rule is proper iff there is a convex function $G$ on $\mathcal P$ with $S(P,\omega)=G(P)-\int G^*(P,\cdot)\,dP+G^*(P,\omega)$, where $G^*(P,\cdot)$ is a subtangent of $G$ at $P$; equivalently $G(P)=S(P,P)$ (the generalized entropy / information measure) is convex and $S(P,\cdot)$ is a subtangent. The associated divergence $d(P,Q)=S(Q,Q)-S(P,Q)$ is a Bregman divergence.
• Categorical scores & the Savage/Schervish representations. For a probability vector $p$ over $m$ categories: the logarithmic score $S(p,i)=\log p_i$ (→ Shannon entropy, KL divergence; Good 1952); the quadratic / Brier score $S(p,i)=2p_i-\sum_j p_j^2-1$ (→ squared-Euclidean Bregman divergence; Brier 1950); the spherical/pseudospherical score. The paper proves a rigorous version of the Savage (1971) representation and connects it to Schervish (1989), writing every proper score for a dichotomous event as a mixture of cost-weighted zero–one scores over cost–loss ratios $c\in(0,1)$.
• Continuous ranked probability score (CRPS). For a predictive CDF $F$ and outcome $x$, $\mathrm{CRPS}(F,x)=\int_{-\infty}^{\infty}\big(F(y)-\mathbf 1\{y\ge x\}\big)^2\,dy = E_F|X-x|-\tfrac12 E_F|X-X'|$. It is proper, reported in the same units as $x$, generalizes the absolute error (to which it reduces for a point forecast), and so lets deterministic and probabilistic forecasts be compared on one scale.
• Energy score (multivariate generalization). $\mathrm{ES}(P,x)=\tfrac12 E_P\lVert X-X'\rVert^{\beta}-E_P\lVert X-x\rVert^{\beta}$ for $\beta\in(0,2)$, with $X,X'$ i.i.d. $\sim P$; strictly proper (Székely 2003). It reduces to the CRPS at $\beta=1,\,m=1$ and admits a kernel representation in terms of negative-definite functions (Hoeffding-type inequalities).
• Quantile and interval scores. Proper scores exist for predictive quantiles; the class is larger than Cervera–Muñoz (1996) conjectured. For a central $(1-\alpha)$ prediction interval $[l,u]$, the (negatively oriented) interval score is $S^{\mathrm{int}}_\alpha(l,u;x)=(u-l)+\tfrac2\alpha(l-x)\mathbf 1\{x<l\}+\tfrac2\alpha(x-u)\mathbf 1\{x>u\}$ — it rewards narrow intervals but penalizes outcomes falling outside, addressing width and coverage jointly.
• Log score = log Bayes factor. In the no-parameter case, $\log B_{12}=\mathrm{LogS}(H_1,X)-\mathrm{LogS}(H_2,X)$ — Good's (1952) "weight of evidence." This ties proper scoring to Bayesian model comparison and shows Bayes factors apply to any probabilistic forecasting rule, not just parametric models. The paper also proposes random-fold cross-validation.
• Optimum score estimation. Minimizing the mean score $S_n(\theta)=\tfrac1n\sum_i S(P_\theta,X_i)$ over $\theta$ gives a consistent estimator; this minimum-contrast / M-estimation family has maximum likelihood as the special case of the logarithmic score, and lets one tailor the loss to the forecasting goal (e.g. the interval score for interval estimation).
• Propriety matters in practice. The Pacific-Northwest weather case study exhibits an improper score under which a forecaster is rewarded for not reporting his true predictive distribution — a concrete cautionary tale for forecast verification.

Concepts Introduced or Extended

• Proper Scoring Rule
• Bayes Factor
• Model Comparison
• Volatility Forecast Evaluation

Entities Mentioned

• Tilmann Gneiting
• Adrian E. Raftery

Quotes

"A scoring rule ... is proper if the forecaster maximizes the expected score for an observation drawn from the distribution $F$ if he or she issues the probabilistic forecast $F$, rather than $G\ne F$. It is strictly proper if the maximum is unique."

"Maximum likelihood estimation forms a special case of optimum score estimation, and optimum score estimation forms a special case of M-estimation."

My Take

This is the reference that turned a scattered meteorology-and-decision-theory folklore into a clean, general theory, and its vocabulary (proper, strictly proper, CRPS, energy score, sharpness-subject-to-calibration) is now standard in Bayesian and econometric forecasting. For this wiki its most useful throughlines are two: (1) the log score = log Bayes factor identity, which makes Bayes-factor model comparison a special case of proper scoring and connects predictive evaluation to marginal-likelihood machinery; and (2) the CRPS/interval score as robust, honesty-enforcing alternatives to the raw MSE/coverage diagnostics used in volatility forecast evaluation and VaR backtesting. The propriety lens also clarifies why the QLIKE/MSE robustness results (Patton 2011) matter: an improper or proxy-sensitive loss can silently reward the wrong model.