Chib-Jeliazkov (2001) Marginal Likelihood from the Metropolis-Hastings Output

Summary

Chib and Jeliazkov (2001) extend the Chib (1995) marginal likelihood identity to samplers containing Metropolis-Hastings (M-H) steps. Where Chib (1995) estimates the posterior ordinate via Rao-Blackwellization of Gibbs full conditionals, the Chib-Jeliazkov (CJ) estimator exploits the local reversibility of M-H subkernels to express the posterior ordinate as a ratio of two expectations — one over the main Markov chain Monte Carlo (MCMC) output, another over a short cheap reduced run at a fixed high-density point. The estimator applies universally to any MCMC scheme as long as the M-H acceptance probabilities are recorded, and extends naturally to multi-block samplers with latent variables.

Key Claims

• Marginal likelihood identity (eq. 3/4): $\log m(y) = \log f(y|\theta^*) + \log\pi(\theta^*) - \log\pi(\theta^*|y)$; reduces to estimating the posterior ordinate at a single high-density point $\theta^*$.
• Limitation of Chib (1995): the Rao-Blackwell estimator for $\pi(\theta^*|y)$ requires tractable full-conditional densities — fails when any parameter block uses a M-H step.
• Local reversibility of M-H subkernel (eq. 7): $p(\theta, \theta^*|y)\pi(\theta|y) = \pi(\theta^*|y)p(\theta^*, \theta|y)$; integrating over $\theta$ gives the CJ representation.
• CJ one-block estimator (eq. 9): $\hat\pi(\theta^*|y) = E_1\{\alpha(\theta,\theta^*|y)q(\theta,\theta^*|y)\} / E_2\{\alpha(\theta^*,\theta|y)\}$; $E_1$ estimated from the full MCMC run $\{\theta^{(g)}\}$; $E_2$ estimated from $J$ draws from proposal $q(\theta^*,\cdot|y)$ in a reduced run at fixed $\theta^*$.
• Log marginal likelihood (eq. 10): $\log\hat{m}(y) = \log f(y|\theta^*) + \log\pi(\theta^*) - \log\hat\pi(\theta^*|y)$.
• Multi-block with latent variables (eq. 11–15): for a two-block case with latent data $z$, the reduced conditional $\hat\pi(\theta_1^*|y)$ uses expectations over $(\theta_2, z)$ from a reduced run; placing the M-H block first avoids M-H steps during the reduced run.
• General $B$-block formula (eq. 16–18): $\log\hat{m}(y) = \log f(y|\theta^*) + \log\pi(\theta^*) - \sum_{i=1}^B \log\hat\pi(\theta_i^*|y,\theta_1^*,\ldots,\theta_{i-1}^*)$; each ordinate uses Rao-Blackwell for Gibbs blocks and CJ for M-H blocks.
• Numerical standard error (NSE) via Delta method (§2.4): vector process $h^{(g)}$ collects all quantities entering $\log\hat{m}(y)$; $\text{var}(\hat{h})$ estimated by Newey-West with $m=40$ lags; NSE from $\sqrt{a'\,\text{var}(\hat{h})\,a}$; validated by 50 independent replications.
• Efficiency principle: more efficient MCMC schemes (lower inefficiency factors) yield smaller NSE for the marginal likelihood estimate; confirmed across all four examples.
• Binary logit (Mroz 1987, $n=200$, 8 parameters, Table 1): tailored proposal NSE ≈ 10× smaller than random-walk proposal; $\log\hat{m}(y) = -144.756$.
• Gaussian longitudinal (AIDS CD4 counts, 467 subjects, Table 2): multi-block Gibbs and single-block M-H samplers give identical point estimates $\log\hat{m} = -3577.57$; NSE differences track sampler inefficiency factors.
• Poisson longitudinal (epileptic seizures, $n=58$): CJ $-915.23$ vs. kernel density $-915.49$; both NSE $\approx 0.1$.
• Multivariate probit (Panel Study of Income Dynamics, PSID, 520×7, 21 correlation parameters): one-$\sigma$-block $-1550.533$ (NSE 0.842) vs. four-$\sigma$-blocks $-1550.45$ (NSE 0.278); single likelihood evaluation ≈ 2 min on 550 MHz CPU — Gelfand-Dey / Meng-Wong importance sampling is prohibitively expensive here.

Concepts Introduced or Extended

• Marginal Data Density
• Metropolis-Hastings Algorithm
• Gibbs Sampler

Entities Mentioned

• Siddhartha Chib
• Ivan Jeliazkov

Quotes

"The problem of computing the marginal likelihood... is solved by noting that the Metropolis-Hastings kernel satisfies a local reversibility condition."

My Take

CJ is arguably the most consequential extension of Chib (1995) because it removes the Gibbs-only restriction that had limited the original method to tractable-conditional models. The estimator is elegant — two cheap simulation averages — and it adds almost nothing to the computational budget when the main MCMC run already records acceptance probabilities. The multivariate probit example (21-dimensional correlation block) demonstrates the method's comparative advantage most sharply: kernel density estimation is biased in high dimensions while CJ delivers NSE < 1. The four-block vs. one-block comparison also shows that sampler efficiency translates directly into marginal likelihood precision — a practical argument for investing in well-tuned proposals.