Interval Forecast Evaluation

Definition

Interval forecast evaluation assesses whether a sequence of prediction intervals (or, equivalently, value-at-risk quantiles) is correctly calibrated. The modern standard, due to Christoffersen (1998), evaluates a $(1-p)$ interval forecast through its hit sequence $I_t=\mathbf 1\{\text{outcome falls in the interval}\}$ and requires correct conditional coverage: the hits must be independent and identically distributed Bernoulli($p$) — the right violation rate and no predictability of violations from the past.

Key Ideas

• Reduce to a 0/1 hit sequence. However complex the forecasting model, its interval forecasts are judged entirely through the binary sequence $I_t$ (inside/outside, or VaR not-exceeded/exceeded). This makes evaluation model-free.
• Unconditional vs. conditional coverage. Unconditional coverage asks only that $\mathbb E[I_t]=p$ (correct average violation rate). Conditional coverage additionally requires the violations to be unpredictable — crucial under time-varying volatility, where a model can have correct average coverage yet cluster its violations in turbulent periods.
• Three likelihood-ratio tests.
  • $LR_{uc}$ — unconditional coverage: empirical hit frequency $=p$.
  • $LR_{ind}$ — independence: modeling $\{I_t\}$ as a first-order Markov chain, test $\pi_{01}=\pi_{11}$ (a violation today is no more likely after a violation yesterday), rejecting clustering.
  • $LR_{cc}$ — conditional coverage: the joint test, with the exact decomposition $LR_{cc}=LR_{uc}+LR_{ind}$ (asymptotically $\chi^2_2$).
• Diagnostic decomposition. Because $LR_{cc}$ splits into a coverage part and an independence part, a rejection tells you how the forecaster fails — wrong average rate, clustered violations, or both.

How It Works

1. Collect out-of-sample realizations and the corresponding $(1-p)$ interval / VaR forecasts; construct the hit sequence $I_t$.
2. Estimate the empirical violation rate and the first-order Markov transition probabilities of $\{I_t\}$.
3. Compute $LR_{uc}$, $LR_{ind}$, and $LR_{cc}$ against their $\chi^2$ reference distributions.
4. Interpret: correct coverage but rejected independence signals a model that ignores volatility dynamics; rejected coverage signals a mis-scaled interval.

Why It Matters

• The VaR backtesting standard. $LR_{cc}$ is the workhorse test for validating VaR models and underlies regulatory and practical backtesting; it is exactly the tool that exposes the HS-VaR pathology of clustered violations after volatility spikes.
• Forces honest interval forecasts. By penalizing predictable violations, it rewards forecasters who let their intervals widen and narrow with conditional volatility rather than reporting a constant-width band.
• Model-free and simple. Requiring only the hit sequence, it applies to any interval forecast regardless of how it was produced.

Open Questions

• Power of the independence test. The first-order-Markov alternative has limited power against richer forms of violation dependence; later tests (Engle-Manganelli's dynamic-quantile / DQ test, Berkowitz density-forecast tests) target more general dynamics.
• Small samples. With few violations (small $p$, short samples) the $\chi^2$ asymptotics are unreliable and finite-sample / Monte Carlo critical values are preferable.
• Beyond coverage. Coverage tests ignore the magnitude of violations; evaluating the full predictive density (or expected shortfall) requires complementary tools.

Related

• Value at Risk
• Volatility Forecast Evaluation
• Coherent Risk Measure
• GARCH and BEKK-GARCH
• Bayesian Multiperiod Prediction
• Peter F. Christoffersen
• Christoffersen (1998) Evaluating Interval Forecasts
• Probability Calibration