Gelfand & Smith (1990) Sampling-Based Approaches to Calculating Marginal Densities

Summary

Gelfand and Smith (1990) introduced the Gibbs sampler to mainstream Bayesian statistics, showing that an algorithm developed for image processing (Geman-Geman 1984) could compute intractable marginal posterior densities in standard statistical models. They present and compare three sampling-based approaches — the substitution/data-augmentation algorithm (Tanner-Wong 1987), the Gibbs sampler, and the Rubin sampling-importance-resampling (SIR) algorithm — across a range of hierarchical and incomplete-data models. The paper is widely credited as triggering the Markov chain Monte Carlo (MCMC) revolution in Bayesian computation.

Key Claims

• Three algorithms unified. Substitution sampling (Tanner-Wong 1987), Gibbs sampling (Geman-Geman 1984), and SIR importance sampling (Rubin 1987) all compute marginal densities without direct integration; the paper provides a unified framework and convergence theory for each.
• Equivalence theorem. When only full conditional distributions are available, substitution sampling and Gibbs sampling are equivalent: the substitution visiting order is just Gibbs sampling with an i.o. (infinitely often) scheme, and Geman-Geman's GG1–GG3 convergence results apply.
• Reduced conditionals can accelerate. When additional reduced conditional distributions are available beyond the k full conditionals, substitution sampling can converge faster; however, empirical experiments show the gain is negligible in practice.
• Rao-Blackwell density estimator. The preferred estimator of $[x]$ uses the full conditional: $[x] \approx \tfrac{1}{m}\sum_{j=1}^m [x \mid Y_j]$, rather than kernel smoothing of the sampled $x$ values. This dominates kernel estimation in mean squared error (MSE) by the Rao-Blackwell theorem.
• SIR inferior to iterative methods. Iterative adaptive sampling (Gibbs/substitution with small m at each step) consistently outperforms a non-iterative SIR draw of equivalent total random-variate budget; the SIR algorithm is highly sensitive to the choice of importance distribution and fails to adapt.
• Worked examples. Multinomial missing-data model (genetic linkage), hierarchical Poisson (pump failures), multivariate normal with missing data, variance components, and errors-in-variables — each yields tractable conjugate full conditionals (Normal / inverse-gamma (IG) / Gamma), making Gibbs straightforward to implement.
• Historical significance. The Gibbs sampler had been confined to image processing and expert systems since Geman-Geman (1984); this paper demonstrated its generality for standard Bayesian posterior computation and launched the modern MCMC era.

Concepts Introduced or Extended

• Gibbs Sampler
• Data Augmentation

Entities Mentioned

• Alan E. Gelfand
• A.F.M. Smith

Quotes

"The attraction of the sampling-based methods is their conceptual simplicity and ease of implementation for users with available computing resources but without numerical analytic expertise."

"All that the user requires is insight into the relevant conditional probability structure and techniques for the efficient generation of appropriate random variates."

My Take

The paper's genius was recognising that Geman-Geman's Gibbs sampler, buried in image processing, solved a general Bayesian problem. The six worked examples collectively showed that full conditionals are often conjugate — Normal, Inverse-Gamma, Gamma — making Gibbs sampling effortless to implement without specialist numerical analysis. The SIR comparison is historically important: iterative adaptive sampling beats one-shot importance sampling for the same computational budget, a lesson the field has repeatedly relearned. The Rao-Blackwell estimator argument is both elegant and practically important — it explains why Gibbs sampling should output conditional density estimates rather than raw samples, and became a cornerstone of subsequent MCMC estimation theory.