Bayesian Model Selection

Definition

Bayesian model selection is the framework for choosing among competing statistical models using probability calculus: either by computing posterior model probabilities via Bayes factors / marginal likelihoods, or by comparing models on how well their predictive distributions describe observed data. Unlike classical hypothesis testing, it provides an interpretable criterion (probability or predictive accuracy) rather than a significance threshold, and naturally handles model uncertainty through averaging.

Key Ideas

Bayes factor: The ratio of marginal likelihoods $B_{10} = p(y|M_1)/p(y|M_0)$ ; summarises all the evidence in data for $M_1$ relative to $M_0$ without requiring a prior on models. Kass-Raftery (1995) provide calibration guidelines.
Posterior model probability: With prior model weights $p(M_k)$ , $p(M_k|y) \propto p(y|M_k)\,p(M_k)$ ; allows coherent model averaging.
Predictive approach (Greenberg-Parks 1997): Instead of computing $p(y|M_k)$ , compare the predictive distribution $p(y_0|X_0, M_k)$ at points of interest. Requires only diffuse priors; avoids specifying prior model probabilities entirely. Particularly useful when the investigator has substantive knowledge about the scale of predictions.
Marginal likelihood identity (Chib 1995): Marginal likelihood = likelihood × prior / posterior, all evaluated at any point (e.g. Maximum Likelihood Estimate, MLE, or posterior mode). Enables computation via Markov Chain Monte Carlo (MCMC) output.
Predictive loss / BPIC (Bayesian Predictive Information Criterion; Ando 2007): Model selection criteria derived from predictive density, penalising complexity via effective number of parameters estimated from MCMC.

How It Works

Bayes Factor Computation

For a regression $y = X\beta + \varepsilon$ , the marginal likelihood integrates out parameters: $p(y|M_k) = \int p(y|\theta_k, M_k)\,p(\theta_k|M_k)\,d\theta_k$ Under Zellner's $g$ -prior ( $\beta \sim N(0, g\sigma^2(X'X)^{-1})$ ), this has a closed form depending only on $R^2$ and degrees of freedom.

Predictive Density Comparison (Greenberg-Parks)

Under a non-informative prior for $(\beta, \sigma)$ , the predictive density for new observations $y_0$ given $X_0$ is multivariate Student-t with mean $X_0\hat\beta$ and covariance proportional to $s^2[I + X_0(X'X)^{-1}X_0']$ . Model comparison proceeds through:

Scaled mean shift $\Lambda'\hat\beta_2/\hat\sigma$ : measures shift in prediction from dropping $X_2$ ; interpretable in units of the dependent variable.
Generalized Variance Ratio (GVR): ratio of determinants of predictive covariance matrices; GVR ≈ 1 means little loss of predictive precision.
Overlap statistic: fraction overlap of 95% highest posterior density (HPD) intervals at sample points across models.

Multicollinearity and Prediction

The mean shift $\Lambda' = X_{02} - X_{01}\hat\delta$ is zero when the prediction point lies on the collinearity hyperplane — the Bayesian restatement of the classical result that multicollinearity does not impair prediction if the regressor correlation structure persists into the forecast period.

Why It Matters

Addresses the well-known failure of null hypothesis testing: with large $n$ , trivially small effects are statistically significant. Bayesian model selection calibrates evidence on the probability scale or on a scale directly relevant to predictions.
Provides a basis for model averaging rather than forced model selection, yielding uncertainty-calibrated forecasts.
The predictive approach (Greenberg-Parks) is especially practical: no marginal likelihood computation, no prior model probabilities, results interpretable by subject-matter experts who understand the scale of the dependent variable.

Open Questions

Bayes factors are sensitive to prior choice under non-informative priors (the Bartlett-Lindley paradox); predictive approaches partially sidestep this but still depend on the prior through the predictive covariance.
Consistency: under mild conditions, posterior model probabilities concentrate on the true model as $n \to \infty$ , but this requires the true model to be in the comparison set.
Computational cost of marginal likelihoods grows with model dimension; MCMC-based estimators (Chib 1995) are feasible but require care to avoid numerical error.