Casella and George (1992) Explaining the Gibbs Sampler

Summary

Casella and George (1992) provide a self-contained tutorial on the Gibbs sampler aimed at statisticians unfamiliar with Markov chain Monte Carlo. They prove convergence via the Markov chain transition matrix in the discrete case and extend the argument to the continuous case via a fixed-point integral equation. The paper establishes conditions under which the algorithm fails (improper joint densities), introduces Rao-Blackwellized density estimation as a variance-reduction device, and discusses practical strategies for extracting samples from the chain.

Key Claims

• The Gibbs sampler generates approximate samples from a marginal $f(x)$ by iterating between conditionals $f(x|y)$ and $f(y|x)$, without ever evaluating $f(x)$ directly.
• The sampled $X$-sequence is a Markov chain with transition matrix $A_{xx} = A_{xy} \cdot A_{yx}$; provided all entries are positive (irreducibility), the unique stationary distribution is the marginal $f_X$ (§3).
• Full conditionals uniquely determine the joint distribution (Besag 1974); the fixed-point integral equation $f_X(x) = \int h(x,t) f_X(t)\,dt$ links them to the marginals, and the Gibbs iteration is its computational solution.
• When the full conditionals do not correspond to any proper joint density, the Gibbs sequence diverges; the canonical failure example is $f(x|y) = ye^{-xy}$, $f(y|x) = xe^{-xy}$, whose fixed-point equation yields the non-integrable $f_X(x) = 1/x$.
• The Rao-Blackwellized density estimator $\hat{f}(x) = m^{-1}\sum_j f(x|Y_j = y_j)$ is unbiased and has lower variance than the empirical histogram estimator (§5); it requires no additional computation once the chain has been run.

Concepts Introduced or Extended

• Gibbs Sampler
• Data Augmentation

Entities Mentioned

• George Casella
• Edward I. George

Quotes

"The Gibbs sampler generates a sample from $f(x)$ using only the conditional distributions $f(x|y)$ and $f(y|x)$."

"In practice, the experimenter does not need $f(x)$, but only needs to be able to calculate $f(x|y)$."

My Take

The paper's main contribution is pedagogical rather than technical: the convergence result via Markov transition matrices and the conditionals-determine-marginals theorem (Besag 1974) were already known. Its lasting value lies in making these ideas accessible, in demonstrating the Rao-Blackwell improvement concretely with both continuous and discrete examples, and in clearly identifying the failure mode from improper conditionals — a warning still underappreciated in practice.