Bayesian Model Averaging

bayesianmodel-averagingmodel-uncertaintybayes-factorvariable-selectionforecasting

Definition

Bayesian model averaging (BMA) accounts for model uncertainty by treating the model itself as an unknown to be integrated out. Given a set of candidate models {M1,…,MK}\{M_1,\ldots,M_K\} and a quantity of interest Δ\Delta (a parameter, a future observation, an effect), the posterior of Δ\Delta is the model-probability-weighted mixture of the model-specific posteriors, p(Δ∣y)=∑k=1Kp(Δ∣Mk,y) p(Mk∣y),p(Mk∣y)=p(y∣Mk) p(Mk)∑lp(y∣Ml) p(Ml),p(\Delta \mid y) = \sum_{k=1}^{K} p(\Delta \mid M_k, y)\, p(M_k \mid y),\qquad p(M_k \mid y) = \frac{p(y \mid M_k)\,p(M_k)}{\sum_{l} p(y \mid M_l)\,p(M_l)}, where p(y∣Mk)=∫p(y∣θk,Mk)p(θk∣Mk) dθkp(y\mid M_k)=\int p(y\mid\theta_k,M_k)p(\theta_k\mid M_k)\,d\theta_k is the marginal likelihood of model kk. Rather than conditioning inference on a single selected model — and thereby understating uncertainty — BMA propagates the analyst's uncertainty about which model is correct into every downstream summary.

Key Ideas

How It Works

  1. Enumerate (or sample) the model space and specify priors p(Mk)p(M_k) and within-model priors p(θk∣Mk)p(\theta_k\mid M_k).
  2. Compute each marginal likelihood p(y∣Mk)p(y\mid M_k) — in closed form under conjugacy, or numerically via bridge sampling, the Chib (1995) identity, or a Laplace/Laplace approximation such as the Laplace–Metropolis estimator of Lewis-Raftery (1997).
  3. Normalise to posterior model probabilities p(Mk∣y)p(M_k\mid y).
  4. Average the model-specific posteriors of Δ\Delta with those weights; report the mixture mean E[Δ∣y]=∑kE[Δ∣Mk,y] p(Mk∣y)\mathrm{E}[\Delta\mid y]=\sum_k \mathrm{E}[\Delta\mid M_k,y]\,p(M_k\mid y) and the mixture variance, which adds a between-model term the single-model variance omits.

Why It Matters

Open Questions

Related