Han-Carlin (2001) MCMC Methods for Computing Bayes Factors: A Comparative Review

Summary

Han and Carlin (2001) review and benchmark six Markov chain Monte Carlo (MCMC) methods for computing Bayes factors across three empirical examples of increasing difficulty. The six methods span product-space samplers (Carlin-Chib 1995, Metropolized CC), reversible jump (Green 1995), partial analytic structure (Godsill 2001), and marginal likelihood identity methods (Chib 1995; Chib-Jeliazkov 2001). The paper finds that product-space samplers fail completely for large hierarchical models while reversible jump and marginal likelihood methods succeed, and recommends marginal likelihood approaches for standard hierarchical models and reversible jump for variable-dimension problems.

Key Claims

• Product Space / Carlin-Chib (CC) (eq. 2): a single Gibbs chain visits all $K$ models simultaneously using pseudo-priors for inactive parameters; model indicator full conditional $p(M=j|\theta,y) \propto f(y|\theta_j)p(\theta_j|M=j)\pi_j$.
• Metropolized CC (MCC) (eq. 4): replaces the full conditional Gibbs step for $M$ with a Metropolis step, drawing from only the proposed-model pseudo-prior; reduces inter-model travel cost for large hierarchical models but inherits CC's scaling failure.
• Reversible Jump (RJ) (Green 1995, eq. 5): dimension-matching move $(\theta_j, u) \mapsto \theta_{j'}$ via bijection $\mathbf{g}_{j,j'}$ with Jacobian $|\partial\mathbf{g}/\partial(\theta_j,u)|$; acceptance ratio $\min[1, \text{likelihood ratio} \times \text{prior ratio} \times \text{Jacobian}]$; handles variable-dimension parameter spaces.
• Partial Analytic Structure (PAS) (Godsill 2001, eq. 6): integrates out analytically tractable subvectors $(\theta_{j'})_\mathcal{U}$ within the RJ acceptance ratio; reduces Monte Carlo variance without extra computation when conjugate conditionals exist.
• Chib (1995) marginal likelihood identity (MLI) (eq. 7): $\log p(y) = \log f(y|\tilde\theta) + \log p(\tilde\theta) - \log p(\tilde\theta|y)$; evaluated at a single high-density point $\tilde\theta$; posterior density estimated by Rao-Blackwellization from Gibbs output.
• Chib-Jeliazkov (CJ 2001) (eq. 11): extends Chib MLI to non-Gibbs samplers via $p(\tilde\theta|y) = E_1\{\alpha(\theta,\tilde\theta|y)q(\theta,\tilde\theta|y)\}/E_2\{\alpha(\tilde\theta,\theta|y)\}$; two-stage Metropolis-Hastings (MH) estimator.
• Radiata pine ($n=42$, Table 2): all six methods work; true $B_{21}=4862$; estimates range from 4782 to 4943 across methods; near-identical performance for this simple linear regression model-comparison problem.
• AIDS CD4 counts (467 subjects, random-effects model): CC and MCC fail completely after 1.5 million iterations — chain mixes within each model but almost never jumps between models; RJ gives $\hat{B}_{21}=67.83$; Chib and CJ also succeed.
• Prostate cancer binary ($n=53$, logistic regression with latent variables): RJ with latent variable augmentation gives $\hat{B}_{21}=6.67$ [6.61, 6.73]; narrow 95% interval demonstrates RJ's efficiency for latent-variable models.
• Main recommendation: marginal likelihood methods (Chib, CJ) preferred for standard hierarchical models; RJ is essential when parameter dimension varies across models; CC/MCC are unreliable for large random-effects problems regardless of run length.

Concepts Introduced or Extended

• Marginal Data Density

Entities Mentioned

• Bradley P. Carlin
• Cong Han
• Siddhartha Chib

Quotes

"When there are many random effects in the model, the product space sampler tends to get 'stuck' in one model and refuse to jump to the other, regardless of the number of MCMC iterations taken." (p. 1128)

My Take

The paper's most important contribution is the negative result: product-space samplers are not a general solution to Bayesian model comparison. Their failure for the 467-subject AIDS model (1.5M iterations, no convergence) is a clear warning against applying CC/MCC to hierarchical models with many latent effects. The RJ method and Chib MLI emerge as the practically useful approaches, with complementary strengths — RJ for dimension-varying problems, Chib for fixed-dimensional standard models. The PAS refinement and CJ extension are incremental improvements; the core message is the failure diagnosis of CC/MCC at scale.