Proper Scoring Rule

Definition

A scoring rule $S(P,x)$ assigns a numerical reward to a probabilistic forecast $P$ (a full predictive distribution) when the outcome $x$ materializes. Writing $S(P,Q)=\int S(P,\omega)\,dQ(\omega)$ for the expected score under $Q$ when $P$ is quoted, the rule is proper if $S(Q,Q)\ge S(P,Q)$ for all $P,Q$ — so a forecaster maximizes expected score by quoting his or her true belief — and strictly proper if equality holds only when $P=Q$ . Propriety is an incentive-compatibility (honesty) property: it removes any reward for hedging, exaggerating sharpness, or otherwise misreporting the predictive distribution.

Key Ideas

Honesty is optimal. A proper rule cannot be gamed: the expected-score-maximizing report is the forecaster's true predictive distribution. Improper but "intuitively appealing" scores can rank a dishonest forecast above an honest one — the failure mode a proper rule is designed to exclude.
Positive orientation. Scores are taken as rewards to be maximized (some literatures report negatively-oriented losses to be minimized — e.g. the interval score below).
Sharpness subject to calibration. Proper scores operationalize the forecasting goal of maximizing the concentration (sharpness) of the predictive distribution subject to its statistical consistency with outcomes (calibration).
Two jobs. Proper rules serve (i) evaluation — ranking competing forecasters or models — and (ii) estimation — as tailored loss/utility functions (optimum-score estimation).

How It Works

Characterization. Every regular proper scoring rule arises from a convex "generalized entropy" function $G(P)=S(P,P)$ via a subtangent $G^*$ : $S(P,\omega)=G(P)-\int G^*(P,\cdot)\,dP+G^*(P,\omega).$ The induced divergence $d(P,Q)=S(Q,Q)-S(P,Q)\ge 0$ is a Bregman divergence of $G$ . Different convex $G$ 's generate the standard rules:

Logarithmic score $S(p,i)=\log p_i$ → Shannon entropy, Kullback–Leibler divergence (Good 1952). Local and strictly proper, but unbounded.
Quadratic / Brier score $S(p,i)=2p_i-\sum_j p_j^2-1$ → squared-Euclidean Bregman divergence (Brier 1950). Introduced to score probabilistic weather forecasts, the Brier score is the mean squared error of the probability forecasts — for a binary event, $\mathrm{BS}=\frac1N\sum_i (f_i-o_i)^2$ with forecast probability $f_i$ and outcome $o_i\in\{0,1\}$ — the first and still most-used proper scoring rule, later decomposed (Murphy) into reliability, resolution, and uncertainty.
Spherical / pseudospherical score → $\ell_\alpha$ -type entropy.
For a dichotomous event, the Savage (1971) and Schervish (1989) representations write any proper score as a mixture of cost-weighted zero–one scores over cost–loss ratios.

Continuous outcomes. The continuous ranked probability score for a predictive CDF $F$ is $\mathrm{CRPS}(F,x)=\int_{-\infty}^{\infty}\big(F(y)-\mathbf 1\{y\ge x\}\big)^2\,dy = E_F|X-x|-\tfrac12 E_F|X-X'|,$ which is reported in the units of $x$ and reduces to absolute error for a point forecast. Its multivariate generalization is the energy score $\mathrm{ES}(P,x)=\tfrac12 E_P\lVert X-X'\rVert^{\beta}-E_P\lVert X-x\rVert^{\beta}$ , $\beta\in(0,2)$ (Székely 2003; CRPS is $\beta=1,m=1$ ).

Interval forecasts. For a central $(1-\alpha)$ prediction interval $[l,u]$ the (negatively oriented) interval score is $S^{\mathrm{int}}_\alpha(l,u;x)=(u-l)+\tfrac2\alpha(l-x)\mathbf 1\{x<l\}+\tfrac2\alpha(x-u)\mathbf 1\{x>u\},$ rewarding narrow intervals but penalizing outcomes that fall outside — width and coverage in one number.

Why It Matters

Forecast verification. Proper scores (especially CRPS and the interval score) are the standard way to rank probabilistic forecasts in weather, macro, and finance, replacing ad-hoc diagnostics that can reward the wrong model.
Bayesian model comparison. In the no-parameter case the log score equals the log Bayes factor — Good's "weight of evidence" — so proper scoring subsumes Bayes-factor model comparison and extends it to any forecasting rule.
Estimation. Optimum-score / minimum-contrast estimation, $\hat\theta=\arg\max_\theta \tfrac1n\sum_i S(P_\theta,X_i)$ , is M-estimation that generalizes maximum likelihood (the log-score case), letting the estimation loss be tailored to the forecasting objective.
Risk applications. Quantile and interval scores grade the predictive quantiles used in value-at-risk and prediction-interval reporting.

Open Questions