Diebold-Mariano Test

Definition

The Diebold-Mariano (DM) test is a test of the null hypothesis of equal out-of-sample predictive accuracy between two forecasts, based only on the observed sequence of forecast errors and an arbitrary loss function (Diebold-Mariano 1995). It works directly with the loss differential $d_t=g(e_{1t})-g(e_{2t})$ and tests whether its population mean is zero — imposing no requirement that the forecasts be nested, the loss quadratic, or the errors Gaussian.

Key Ideas

• Loss differential. For two competing forecasts with errors $e_{1t},e_{2t}$ and a loss $g(\cdot)$, define $d_t=g(e_{1t})-g(e_{2t})$. Equal accuracy is the null $H_0:E[d_t]=0$.
• Generality. The loss need not be quadratic or even symmetric (any $g$, including direction-of-change), and the errors may be non-Gaussian, non-zero-mean, serially correlated, and contemporaneously correlated across the two forecasts — the setting where earlier tests (Morgan-Granger-Newbold) fail.
• Asymptotic statistic. $\displaystyle S_1=\frac{\bar d}{\sqrt{2\pi \hat f_d(0)/T}}\xrightarrow{d}N(0,1)$, where $\bar d$ is the mean loss differential and $2\pi f_d(0)=\sum_{\tau=-\infty}^{\infty}\gamma_d(\tau)$ is the (long-run) spectral density of $d_t$ at frequency zero. Estimating $f_d(0)$ from a weighted sum of sample autocovariances corrects for the serial correlation that $h$-step-ahead errors inevitably carry (truncating at $h-1$ lags).
• Exact finite-sample variants. For i.i.d. loss differentials, the nonparametric sign test (null: zero median loss differential, $P(g(e_1)>g(e_2))=\tfrac12$) and Wilcoxon signed-rank test (using ranks) give exact tests; under a symmetric loss differential the zero-median and zero-mean nulls coincide.

Why It Matters

• The standard out-of-sample comparison. DM became the default way to ask "is forecast A significantly more accurate than forecast B?" across economics and finance — used throughout this wiki for point-forecast comparison (see Forecasting, Model Comparison, Volatility Forecast Evaluation).
• Loss-function agnostic. Because any loss works, the test aligns evaluation with the decision problem (asymmetric loss, direction-of-change, economic value) rather than forcing MSE.
• No nesting required. Unlike in-sample F/likelihood-ratio tests, DM compares any two forecast sequences, nested or not — though the nested case needs care (West's and Clark-McCracken's out-of-sample-inference refinements).

Open Questions

• Nested models and parameter estimation. When forecasts come from nested models with estimated parameters, the DM limiting distribution can be non-standard; West (1996) and Clark-McCracken adjust the inference.
• Small samples. The asymptotic $S_1$ can over-reject in small out-of-sample windows; the Harvey-Leybourne-Newbold size correction and the finite-sample sign/Wilcoxon tests mitigate this.
• Choice of long-run variance estimator (kernel, bandwidth) materially affects the test.

Related

• Forecasting
• Model Comparison
• Volatility Forecast Evaluation
• Proper Scoring Rule
• Diebold-Mariano (1995) Comparing Predictive Accuracy
• Francis X. Diebold
• Roberto S. Mariano