Estimating the Expected Predictive Accuracy of Econometric Models

Summary

Fair (1980) proposes a stochastic-simulation framework that decomposes forecast uncertainty into four hierarchically nested components: randomness in the error terms, uncertainty in the coefficient estimates, uncertainty in the exogenous-variable forecasts, and model misspecification. The total forecast variance is the sum of the stochastic-simulation variance and a misspecification correction estimated from outside-sample residuals. Applied to a large macroeconometric model (97 equations) and a naive eighth-order autoregressive (AR(8)) benchmark over 1978II–1981IV, the method yields error bounds with an explicit structural interpretation unavailable from root-mean-square error (RMSE) statistics alone.

Key Claims

Forecast uncertainty has four distinct sources: (a) stochastic error terms, (b) coefficient-estimate uncertainty, (c) exogenous-variable-forecast uncertainty, and (d) model misspecification.
The four components are nested: adding each component in sequence yields $\sigma^2_a \leq \sigma^2_b \leq \sigma^2_c \leq \tilde\sigma^2(x)$ , so the relative magnitude of each source is directly readable.
The misspecification component $d_i(t,k,\ldots) = \hat{e}^2_i(\text{outside-sample}) - \tilde\sigma^2_i(\text{stochastic-sim})$ is estimated by rolling reestimation (35 iterations for the 1954I–1977IV sample), under the constancy assumption that its mean is stable over time.
Total variance without trend: $\tilde\sigma^2(x) + \bar{d}_i(k)$ [eq. 4]; with trend: $\tilde\sigma^2(x) + \bar{d}_i(k)\cdot\bar{y}^2_i$ [eq. 5].
Model I (Fair macro: 97 equations, 29 stochastic, 183 coefficients, 60 exogenous variables, 1000 trials) dominates Model II (naive AR(8), 2000 trials) on 4 of 6 variables at 8-quarter-ahead horizons.
RMSE is an inferior evaluation criterion because it conflates all uncertainty sources and cannot separate structural forecast error from simulation variance.
The constancy assumption — that the misspecification component has a constant mean across time — is acknowledged as strong but untestable without additional data; it is the method's main vulnerability.
Exogenous-variable uncertainty (component c) is often the largest single contributor to total variance in large macro models, exceeding coefficient and residual uncertainty.
The method is computationally demanding: each of the 35 rolling reestimations requires a full model solve plus stochastic simulation, making it expensive for models of Model I's size.

Concepts Introduced or Extended

Entities Mentioned

Quotes

"The main purpose of this paper is to propose a method of estimating the expected predictive accuracy of an econometric model, where expected predictive accuracy is measured by the expected squared error of each endogenous variable forecast." (p. 355)

"The constancy assumption is clearly a strong one. It assumes, for example, that the model is not getting better or worse over time relative to a perfect model." (p. 362)

"One of the advantages of the present method over the use of root mean square errors is that the present method does not require one to wait until the forecast period is over." (p. 374)

My Take

Fair's decomposition is methodologically elegant: the nested structure of the four components gives a direct diagnostic of where forecast error comes from — something RMSE cannot provide. The constancy assumption for the misspecification component is the Achilles heel; if model quality drifts (structural breaks, omitted variables entering relevance over time), the estimated $\bar{d}_i(k)$ will be biased. The rolling-reestimation approach (35 iterations) is reasonable for 1980 but the computational cost scales badly. The result that exogenous-variable uncertainty often dominates is practically important and underappreciated in evaluation exercises that treat exogenous inputs as known. The comparison with a naive AR(8) also anticipates the forecast-combination and benchmark-model literature that would flourish in the 1980s–1990s.