Hoeting-Madigan-Raftery-Volinsky (1999) Bayesian Model Averaging: A Tutorial

Summary

The canonical tutorial on Bayesian model averaging (BMA) — the coherent way to account for model uncertainty rather than conditioning all inference on a single selected model. Standard practice picks one model and proceeds as if it generated the data, which understates uncertainty and yields over-confident inferences and riskier decisions. BMA instead averages the quantity of interest over all candidate models, weighted by their posterior probabilities. The paper lays out the machinery, the two computational bottlenecks (summing over models, computing each model's integral) and their solutions (Occam's window, MC³, Laplace), implementation for specific model classes (linear regression, GLMs, survival, graphical models), prior specification, and empirical evidence that BMA improves out-of-sample predictive performance.

Key Claims

• The BMA identity. For a quantity of interest $\Delta$ (effect size, future observable, utility) and models $M_1,\dots,M_K$: $$\Pr(\Delta\mid D)=\sum_{k=1}^{K}\Pr(\Delta\mid M_k,D)\,\Pr(M_k\mid D),\qquad \Pr(M_k\mid D)=\frac{\Pr(D\mid M_k)\Pr(M_k)}{\sum_l \Pr(D\mid M_l)\Pr(M_l)},$$ with $\Pr(D\mid M_k)=\int \Pr(D\mid\theta_k,M_k)\Pr(\theta_k\mid M_k)\,d\theta_k$ the integrated (marginal) likelihood.
• Mean/variance decomposition. $E[\Delta\mid D]=\sum_k\hat\Delta_k\Pr(M_k\mid D)$ and $\mathrm{Var}[\Delta\mid D]=\sum_k(\mathrm{Var}[\Delta\mid D,M_k]+\hat\Delta_k^2)\Pr(M_k\mid D)-E[\Delta\mid D]^2$ — the posterior variance includes between-model variance that single-model inference ignores.
• Predictive optimality. Averaging beats any single model $M_j$ under a logarithmic scoring rule (Madigan-Raftery 1994), a theoretical claim backed by extensive empirical evidence.
• Managing the summation — Occam's window (Madigan-Raftery 1994). Discard models predicting far worse than the best (posterior-odds ratio $> C$), and optionally (Occam's razor) discard complex models dominated by a simpler nested submodel; leaves a small set $\mathcal A$ (often <100, sometimes 1–2 models). Volinsky et al. (1997) use "leaps and bounds" to find the retained models fast.
• Managing the summation — MC³ (Madigan-York 1995). Markov chain Monte Carlo model composition: build a Markov chain over model space with a neighborhood proposal (add/drop one term) and Metropolis acceptance whose stationary distribution is $\Pr(M\mid D)$; average $g(M)=\Pr(\Delta\mid M,D)$ over the chain.
• Computing the integrals. Closed form under conjugacy; otherwise Laplace approximation (and the Laplace-Metropolis estimator), or MCMC — the BIC/Bayes-factor approximations of Kass-Raftery (1995) for screening.
• Broad applicability + software. Worked recipes for linear regression (predictors, outliers, transformations), generalized linear models, Cox/survival analysis, and graphical models with missing data; a catalogue of BMA software (the BMA toolkit). Two applied examples (primary biliary cirrhosis survival; body-fat prediction) show improved calibration and predictive scores.

Concepts Introduced or Extended

• Bayesian Model Averaging

Entities Mentioned

• Jennifer A. Hoeting
• David Madigan
• Adrian E. Raftery
• Chris T. Volinsky

Quotes

"Data analysts typically select a model from some class of models and then proceed as if the selected model had generated the data. This approach ignores the uncertainty in model selection, leading to over-confident inferences and decisions that are more risky than one thinks they are."

"Bayesian model averaging (BMA) provides a coherent mechanism for accounting for this model uncertainty."

My Take

This is the reference everyone cites for BMA, and rightly so: it is less a research paper than a well-built manual that turns a philosophically obvious idea — don't condition on a model you're not sure of — into concrete, class-by-class recipes plus honest attention to the two things that actually stop people using it (the sum over $2^p$ models and the per-model integral). The lasting conceptual takeaways for the wiki are the variance decomposition (single-model standard errors are too small because they omit between-model variance) and the two complementary scaling strategies: Occam's window (prune to a data-supported handful) versus MC³ (random-walk the model space). It sits directly on top of the marginal-likelihood machinery elsewhere in the wiki — marginal data density, Chib's identity, bridge sampling, the Laplace-Metropolis estimator — since every model weight is a marginal likelihood, and it is the natural umbrella over spike-and-slab variable selection and stochastic-search model methods. The one caveat it is candid about is that BMA answers "average over this model class"; choosing the class, and whether any of it is remotely true, remains the analyst's judgment.