Bayesian Model Averaging

Definition

Bayesian model averaging (BMA) accounts for model uncertainty by treating the model itself as an unknown to be integrated out. Given a set of candidate models $\{M_1,\ldots,M_K\}$ and a quantity of interest $\Delta$ (a parameter, a future observation, an effect), the posterior of $\Delta$ is the model-probability-weighted mixture of the model-specific posteriors, $$p(\Delta \mid y) = \sum_{k=1}^{K} p(\Delta \mid M_k, y)\, p(M_k \mid y),\qquad p(M_k \mid y) = \frac{p(y \mid M_k)\,p(M_k)}{\sum_{l} p(y \mid M_l)\,p(M_l)},$$ where $p(y\mid M_k)=\int p(y\mid\theta_k,M_k)p(\theta_k\mid M_k)\,d\theta_k$ is the marginal likelihood of model $k$. Rather than conditioning inference on a single selected model — and thereby understating uncertainty — BMA propagates the analyst's uncertainty about which model is correct into every downstream summary.

Key Ideas

• Posterior model probabilities are the weights. Each model's weight is proportional to its marginal likelihood times its prior probability; pairwise weights are governed by the Bayes factor $B_{kl}=p(y\mid M_k)/p(y\mid M_l)$. This is the same ingredient as Bayesian model selection — selection takes the argmax, averaging takes the full mixture.
• Averaging beats selection for prediction. BMA minimises expected logarithmic (and, under a proper scoring rule, other) predictive loss; a large empirical literature finds averaged forecasts are better calibrated than any single "winner," echoing the forecast-combination finding that pooling dominates picking.
• Model-space priors and Occam's window. With $2^p$ models in variable selection the sum is intractable; practical BMA either uses MCMC over model space (spike-and-slab / MC$^3$) or restricts to an "Occam's window" of models within a likelihood factor of the best, discarding the long tail of near-zero-weight models. Raftery-Madigan-Hoeting (1997) is the original development of both strategies for linear regression, later distilled by the Hoeting-Madigan-Raftery-Volinsky (1999) tutorial: Occam's window (Madigan-Raftery 1994) prunes the model set with a two-part rule — drop models predicting far worse than the best (posterior-odds ratio $>C$) and, optionally, complex models dominated by a simpler nested submodel — typically leaving fewer than 100 (often <10) models; MC³ (Markov chain Monte Carlo model composition; Madigan-York 1995) runs a Metropolis chain over model space with an add/drop-one-term neighborhood proposal whose stationary distribution is $p(M\mid D)$. A striking property they document: in the null case (many candidate predictors, none truly related to the response) — where ordinary variable selection routinely returns a spuriously high-$R^2$, significant-$F$ subset — Occam's window instead concentrates on the null model, largely dissolving the false-discovery problem.
• Priors matter more than in single-model inference. Because the marginal likelihood is sensitive to the within-model prior spread (Lindley/Bartlett effect), diffuse parameter priors can silently penalise larger models; unit-information or $g$-priors are common defaults.
• Distinct from continuous shrinkage. BMA mixes over discrete models; horseshoe and Lasso priors instead shrink within one encompassing model. The two can be combined but answer different questions.

How It Works

1. Enumerate (or sample) the model space and specify priors $p(M_k)$ and within-model priors $p(\theta_k\mid M_k)$.
2. Compute each marginal likelihood $p(y\mid M_k)$ — in closed form under conjugacy, or numerically via bridge sampling, the Chib (1995) identity, or a Laplace/Laplace approximation such as the Laplace–Metropolis estimator of Lewis-Raftery (1997).
3. Normalise to posterior model probabilities $p(M_k\mid y)$.
4. Average the model-specific posteriors of $\Delta$ with those weights; report the mixture mean $\mathrm{E}[\Delta\mid y]=\sum_k \mathrm{E}[\Delta\mid M_k,y]\,p(M_k\mid y)$ and the mixture variance, which adds a between-model term the single-model variance omits.

Why It Matters

• Honest uncertainty. The between-model variance term is exactly the uncertainty that model selection throws away; ignoring it produces overconfident intervals.
• Ubiquitous in applied Bayesian econometrics. BMA over regressor subsets is standard in cross-country growth regressions, over lag length and variable inclusion in Bayesian VARs, and over cointegration rank $r$ in cointegration analysis (Koop-Potter-Strachan 2005).
• A byproduct of variable-selection samplers. The posterior inclusion probabilities produced by spike-and-slab Gibbs samplers are BMA weights for the "include/exclude" decision, so BMA often comes for free once a selection prior is in place.

Open Questions

• Prior sensitivity of weights. Marginal likelihoods can be dominated by prior spread; how to set within-model priors so that averaging is robust remains contested (mixtures of $g$-priors are one answer).
• $\mathcal{M}$-open settings. When the true data-generating process is in none of the candidates, BMA weights concentrate on the single Kullback–Leibler-closest model asymptotically; predictive stacking can outperform BMA here.
• Computation at scale. For very large model spaces, MCMC over models mixes poorly; scalable approximations (variational, screened Occam's window) trade exactness for feasibility.

Related

• Bayesian Model Selection
• Model Comparison
• Bayes Factor
• Marginal Data Density
• Forecast Combination
• Variable Selection
• Spike-and-Slab Prior
• Bridge Sampling
• Chen-Ibrahim-Yiannoutsos (1999) Prior Elicitation, Variable Selection and Bayesian Computation for Logistic Regression Models
• Lewis-Raftery (1997) Estimating Bayes Factors via Posterior Simulation with the Laplace-Metropolis Estimator
• Hoeting-Madigan-Raftery-Volinsky (1999) Bayesian Model Averaging: A Tutorial
• Raftery-Madigan-Hoeting (1997) Bayesian Model Averaging for Linear Regression Models
• Adrian Raftery
• Jennifer A. Hoeting
• David Madigan
• Chris T. Volinsky
• Bayesian Adaptive Sampling
• Clyde-Ghosh-Littman (2011) Bayesian Adaptive Sampling for Variable Selection and Model Averaging
• Merlise Clyde