Diebold-Mariano (1995) Comparing Predictive Accuracy

Summary

Diebold and Mariano propose explicit tests of the null hypothesis of no difference in the accuracy of two competing forecasts. Unlike earlier procedures, the tests accommodate a wide variety of accuracy measures — the loss function need not be quadratic or even symmetric — and forecast errors that are non-Gaussian, non-zero-mean, serially correlated, and contemporaneously correlated. They develop an asymptotic test based on the mean loss differential and its long-run variance, plus exact finite-sample sign and Wilcoxon signed-rank tests, and evaluate their size and power.

Key Claims

• Equal-accuracy null via the loss differential. With loss $g(\cdot)$ and forecast errors $e_{1t},e_{2t}$, define $d_t=g(e_{1t})-g(e_{2t})$; equal predictive accuracy is $H_0:E[d_t]=0$. Any loss (asymmetric, direction-of-change, or general $g(y_t,\hat y_t)$) is permitted.
• Asymptotic test. If $d_t$ is covariance-stationary short-memory, $\sqrt T(\bar d-\mu)\to N(0,2\pi f_d(0))$, giving $S_1=\bar d/\sqrt{2\pi \hat f_d(0)/T}\to N(0,1)$; $2\pi f_d(0)=\sum_\tau\gamma_d(\tau)$ is estimated by a weighted sum of sample autocovariances, correcting for the serial correlation of $h$-step errors.
• Exact finite-sample tests. The sign test (null: zero median loss differential) and Wilcoxon signed-rank test (ranks of $|d_t|$) provide exact tests when the loss differential is i.i.d.; under a symmetric loss differential the zero-median null coincides with the zero-mean null.
• Contrast with predecessors. The Morgan-Granger-Newbold approach relies on orthogonalizing transformations and Gaussian, zero-mean, non-autocorrelated errors — assumptions that DM removes.
• Evaluation. Monte Carlo experiments assess finite-sample size and power of the asymptotic and exact tests.

Concepts Introduced or Extended

• Diebold-Mariano Test
• Forecasting

Entities Mentioned

• Francis X. Diebold
• Roberto S. Mariano

Quotes

"We propose and evaluate explicit tests of the null hypothesis of no difference in the accuracy of two competing forecasts. In contrast to previously developed tests, a wide variety of accuracy measures can be used (in particular, the loss function need not be quadratic, and need not even be symmetric), and forecast errors can be non-Gaussian, non-zero mean, serially correlated, and contemporaneously correlated."

My Take

This short paper defined how a generation compares forecasts: reduce two competitors to a single loss-differential series and test its mean, with a long-run-variance correction for the serial dependence that multi-step errors always have. Its lasting strength is loss-function agnosticism — evaluation can finally match the decision problem instead of defaulting to MSE. The now-standard caveats came later and are catalogued in the concept page: the asymptotics get delicate for nested models with estimated parameters (West 1996; Clark-McCracken) and the $S_1$ statistic over-rejects in small windows (Harvey-Leybourne-Newbold correction). Diebold himself later stressed that DM tests a property of forecasts, not of models — the distinction that keeps its application honest.