Bayes Factor

Definition

The Bayes factor $\text{BF}_{12}$ is the ratio of marginal likelihoods of two models $\mathcal{M}_1$ and $\mathcal{M}_2$:

$$\text{BF}_{12} = \frac{p(y \mid \mathcal{M}_1)}{p(y \mid \mathcal{M}_2)}$$

where $p(y \mid \mathcal{M}_k) = \int L(\theta; y) \, p(\theta \mid \mathcal{M}_k) \, d\theta$ is the marginal data density (evidence) under model $k$. Values $\text{BF}_{12} > 1$ favour $\mathcal{M}_1$; the Jeffreys (1961) scale classifies $\ln \text{BF}_{12} > 1$ as "substantial" and $> 5$ as "decisive."

Key Ideas

• Automatic Occam's razor: the marginal likelihood penalises overparameterised models automatically through the prior predictive — models with high flexibility are penalised for spreading prior mass over unlikely data regions.
• No free parameters: unlike Akaike Information Criterion (AIC)/Bayesian Information Criterion (BIC), Bayes factors make no asymptotic approximation; they are exact given the prior.
• Prior sensitivity: since $\text{BF}_{12}$ depends on the prior, improper or vague priors make $\text{BF}_{12}$ undefined or Bartlett-paradox-prone; proper priors are required.
• Posterior model probabilities: given prior model probabilities $P(\mathcal{M}_k)$, the posterior odds equal prior odds times Bayes factor: $P(\mathcal{M}_1|y)/P(\mathcal{M}_2|y) = \text{BF}_{12} \cdot P(\mathcal{M}_1)/P(\mathcal{M}_2)$.

How It Works

• Closed form: available for conjugate models (Normal-Wishart Vector Autoregression, VAR); see Marginal Data Density.
• Chib (1995) identity: for Gibbs-sampler output $\log p(y) = \log L(\theta^*|y) + \log p(\theta^*) - \log p(\theta^*|y)$, evaluated at a point estimate.
• Harmonic mean estimator (Newton-Raftery 1994): $\hat p(y) = [G^{-1} \sum_g L(\theta^{(g)};y)^{-1}]^{-1}$ — consistent but has infinite variance in practice.
• Bridge sampling (Meng-Wong 1996): more stable than harmonic mean; recommended for high-dimensional posteriors.
• BIC as an asymptotic Bayes factor (Raftery 1986): under mild conditions $-2\log B_{01}\overset{p}{\simeq}\chi^2 - s\log n = \text{BIC}$ (Schwarz 1978), where $\chi^2$ is the likelihood-ratio statistic, $s$ the number of extra constraints, and $n$ the sample size — so the Schwarz criterion is an $O(1)$ approximation to twice the log Bayes factor that avoids the prior-elicitation and integration problems (later systematized by Kass-Raftery 1995). For log-linear contingency-table models the Spiegelhalter-Smith (1982) approximate Bayes factor is indeterminate at empty cells; replacing their prior with a proper Jeffreys prior (equivalently, adding $\tfrac12$ to each cell count) both removes the zero-cell problem and eliminates the arbitrary constant that vague priors otherwise introduce.

Why It Matters

Bayes factors provide the canonical Bayesian approach to model selection and hypothesis testing, including non-nested comparisons that classical likelihood ratio tests cannot handle.

Open Questions

• Sensitivity to the choice of prior on parameters — particularly important for testing point null hypotheses (the Savage-Dickey density ratio applies when the null is nested).
• The Bayes factor reversal (BFR) paradox (Lovric 2025). Prior sensitivity can flip the direction of the evidence, not merely its magnitude. In the normal model with known variance, reparametrizing by $k=n\tau^2$ (effective prior precision) gives $\mathrm{BF}_{01}(z;k)=\sqrt{1+k}\,\exp\!\big(-\tfrac{z^2k}{2(1+k)}\big)$: at $k=0$ it is $1$ (neutral); as $k\to\infty$ it diverges to $+\infty$ (favouring the null — the Jeffreys–Lindley effect); and for any two-sided result significant at 0.05 there is a flip point in between where the Bayes factor crosses $1$. So the same data testing the same hypothesis can favour the alternative under one prior variance and the null under another, purely from the prior choice — and, unlike the Jeffreys–Lindley paradox which needs $n\to\infty$, this occurs at realistic sample sizes (formalizing Robert's 2016 concern about prior-scale arbitrariness).
• Computational difficulty for large models where Markov Chain Monte Carlo (MCMC) is required to estimate each marginal likelihood.

Related

• Marginal Data Density
• Gibbs Sampler
• Proper Scoring Rule
• Almost Ideal Demand System
• Ordinal Regression
• Bridge Sampling
• Bayesian Model Averaging
• Raftery (1986) A Note on Bayes Factors for Log-linear Contingency Table Models with Vague Prior Information
• Kass-Raftery (1995) Bayes Factors
• Lovric (2025) The Bayes Factor Reversal Paradox
• Adrian E. Raftery
• Robert E. Kass
• Miodrag M. Lovric
• Zellner's g-Prior