Bayes Factor

Definition

The Bayes factor $\text{BF}_{12}$ is the ratio of marginal likelihoods of two models $\mathcal{M}_1$ and $\mathcal{M}_2$:

$$\text{BF}_{12} = \frac{p(y \mid \mathcal{M}_1)}{p(y \mid \mathcal{M}_2)}$$

where $p(y \mid \mathcal{M}_k) = \int L(\theta; y) \, p(\theta \mid \mathcal{M}_k) \, d\theta$ is the marginal data density (evidence) under model $k$. Values $\text{BF}_{12} > 1$ favour $\mathcal{M}_1$; the Jeffreys (1961) scale classifies $\ln \text{BF}_{12} > 1$ as "substantial" and $> 5$ as "decisive."

Key Ideas

• Automatic Occam's razor: the marginal likelihood penalises overparameterised models automatically through the prior predictive — models with high flexibility are penalised for spreading prior mass over unlikely data regions.
• No free parameters: unlike Akaike Information Criterion (AIC)/Bayesian Information Criterion (BIC), Bayes factors make no asymptotic approximation; they are exact given the prior.
• Prior sensitivity: since $\text{BF}_{12}$ depends on the prior, improper or vague priors make $\text{BF}_{12}$ undefined or Bartlett-paradox-prone; proper priors are required.
• Posterior model probabilities: given prior model probabilities $P(\mathcal{M}_k)$, the posterior odds equal prior odds times Bayes factor: $P(\mathcal{M}_1|y)/P(\mathcal{M}_2|y) = \text{BF}_{12} \cdot P(\mathcal{M}_1)/P(\mathcal{M}_2)$.

How It Works

• Closed form: available for conjugate models (Normal-Wishart Vector Autoregression, VAR); see Marginal Data Density.
• Chib (1995) identity: for Gibbs-sampler output $\log p(y) = \log L(\theta^*|y) + \log p(\theta^*) - \log p(\theta^*|y)$, evaluated at a point estimate.
• Harmonic mean estimator (Newton-Raftery 1994): $\hat p(y) = [G^{-1} \sum_g L(\theta^{(g)};y)^{-1}]^{-1}$ — consistent but has infinite variance in practice.
• Bridge sampling (Meng-Wong 1996): more stable than harmonic mean; recommended for high-dimensional posteriors.

Why It Matters

Bayes factors provide the canonical Bayesian approach to model selection and hypothesis testing, including non-nested comparisons that classical likelihood ratio tests cannot handle.

Open Questions

• Sensitivity to the choice of prior on parameters — particularly important for testing point null hypotheses (the Savage-Dickey density ratio applies when the null is nested).
• Computational difficulty for large models where Markov Chain Monte Carlo (MCMC) is required to estimate each marginal likelihood.

Related

• Marginal Data Density
• Gibbs Sampler
• Almost Ideal Demand System
• Ordinal Regression