Raftery-Madigan-Hoeting (1997) Bayesian Model Averaging for Linear Regression Models

Summary

Raftery, Madigan and Hoeting address model uncertainty in linear regression: conditioning inference on a single selected model ignores the uncertainty about which set of predictors is right, and so understates the uncertainty in every downstream quantity. The Bayesian remedy — averaging over all subsets of predictors, weighted by posterior model probabilities — is usually infeasible for $2^p$ models, so they offer two practical routes: Occam's window, which prunes the model space to a small averageable set, and Markov chain Monte Carlo model composition (MC³), which approximates the full average by a Markov chain over model space. Both give better predictive performance than any single reasonably-selected model.

Key Claims

• Model uncertainty must be averaged, not selected away. Picking one "best" subset and reporting its standard errors understates uncertainty; the coherent Bayesian answer averages the quantity of interest over all models with posterior-probability weights.
• Occam's window. Extending Madigan-Raftery's (1994) graphical model-selection algorithm to regression, prune the model set with a two-part rule — discard models far less supported than the best (posterior-odds ratio exceeds a threshold), and optionally discard complex models beaten by a simpler submodel — leaving a small set (often <10) over which to average.
• MC³ (Madigan-York 1995). A Metropolis chain that moves through model space (add/drop one predictor), whose stationary distribution is the posterior over models; time-averaging over the chain approximates the exact BMA solution.
• Better prediction. Under model uncertainty, both averaging procedures beat any single selected model in out-of-sample predictive performance.
• Resolves spurious significance. When many candidate predictors have no real relationship to the response, standard selection often returns a high-$R^2$, significant-$F$ subset; Occam's window instead concentrates on the null model, largely resolving the false-positive problem.

Concepts Introduced or Extended

• Bayesian Model Averaging

Entities Mentioned

• Adrian E. Raftery
• David Madigan
• Jennifer A. Hoeting

Quotes

"Conditioning on a single selected model ignores model uncertainty, and thus leads to the underestimation of uncertainty when making inferences about quantities of interest … we describe an ad hoc procedure, 'Occam's window' … [and] a Markov chain Monte Carlo approach that directly approximates the exact solution."

My Take

This is the paper that turned Bayesian model averaging from an in-principle ideal into a usable regression tool, and its two devices still define the practical toolkit: Occam's window for when you can prune to a handful of models, MC³ for when you cannot. The null-case result is the one worth remembering — that ordinary variable selection manufactures significant predictors out of pure noise, while BMA's honesty about model uncertainty collapses onto the null — because it is the cleanest demonstration of why averaging beats selecting, and it prefigures the same multiple-testing anxieties later formalised in the factor zoo and backtest-overfitting literatures. On the wiki it is the primary source behind the BMA page's Occam's-window/MC³ material, which the 1999 tutorial later popularised and the deflated-Sharpe/BAS lines extended.