Jackman (2000) Estimation and Inference via Bayesian Simulation: An Introduction to Markov Chain Monte Carlo

Summary

A pedagogical tour of Bayesian simulation methods for political scientists, tracing the progression from maximum likelihood estimation (MLE) through the expectation-maximization (EM) algorithm to the Gibbs sampler and Metropolis-Hastings. Working examples include binary probit via Albert-Chib (1993) truncated-Normal data augmentation, AR(1) regression as a 3-block Gibbs sampler, and multinomial probit (MNP) via McCulloch-Polson-Rossi. The paper advocates BUGS/WinBUGS as a near-universal Markov chain Monte Carlo (MCMC) implementation vehicle.

Key Claims

• EM and Gibbs share the same data augmentation structure (Tanner 1996): augment Y with latent Z; EM maximizes E[log p(θ|Y,Z)], Gibbs draws from p(θ|Y,Z). Table 1 contrasts MLE (optimize L), EM (iterate E+M steps to mode), and MCMC (samples from posterior).
• Albert-Chib (1993) truncated-Normal probit: augment each $y_i$ with latent $y^*_i \sim \mathrm{TN}_{(0,\infty)}(x_i'\beta, 1)$ if $y_i=1$ and $\mathrm{TN}_{(-\infty,0)}$ if $y_i=0$; $\beta$ then has a Normal full conditional — the binary probit problem becomes tractable Gibbs.
• Metropolis-Hastings acceptance ratio: $r = [p(\theta^*|Y)\, q(\theta^{t-1}|\theta^*)] / [p(\theta^{t-1}|Y)\, q(\theta^*|\theta^{t-1})]$; accept with $\min(r, 1)$. Reduces to standard Gibbs when $q$ is a full conditional ($r \equiv 1$ everywhere).
• Convergence diagnostics: Geweke (1992) spectral Z-test comparing $n_A = 0.1n$ early and $n_B = 0.5n$ late subchains; Gelman-Rubin $\sqrt{\hat{R}} \leq 1.2$ from $m \geq 2$ overdispersed chains.
• MNP (McCulloch-Polson-Rossi 1994/1998): $\Sigma$ identified by $\sigma_{11}=1$; sampler decomposes $\Sigma$ into scalar $\gamma$ and matrix $\Phi$ with Wishart conditional; 1992 Bush-Clinton-Perot example confirms independence-of-irrelevant-alternatives (IIA) violation.
• BUGS/WinBUGS provides a near-universal MCMC engine: the directed acyclic graph (DAG) representation of the model automatically determines all full conditionals.

Concepts Introduced or Extended

• Gibbs Sampler

Entities Mentioned

• Simon Jackman

Quotes

"The EM algorithm and the Gibbs sampler are closely related: both exploit the notion of data augmentation."

"The posterior density is the fundamental inferential object in Bayesian analysis, and MCMC methods provide a powerful and general means of summarising it."

My Take

A well-crafted entry point for political scientists encountering MCMC. The EM↔Gibbs parallel via data augmentation is genuinely clarifying: both solve the same intractable integral, one by optimization and one by simulation. No new methodology is contributed; value is entirely pedagogical. The MNP section is the densest and most practically useful part. Note: the Albert-Chib (1993) cited here is JASA 88:669–679 on binary and polychotomous probit — distinct from the Albert-Chib (1993) JBES paper on Markov-switching AR already in this wiki.