Damien-Wakefield-Walker (1999) Gibbs Sampling for Bayesian Non-Conjugate and Hierarchical Models by Using Auxiliary Variables

Summary

Damien, Wakefield, and Walker introduce a systematic procedure for sampling non-standard posterior distributions by introducing auxiliary (latent) variables into the Gibbs sampler. The key theorem shows that any density $f(x) \propto \pi(x)\prod_i l_i(x)$ — where $\pi$ is a proper density and each $l_i$ is non-negative and invertible — can be augmented with uniform latent variables $U_i \sim U(0, l_i(x))$, turning every conditional for $U_i$ into a trivial uniform draw and the conditional for $X$ into a draw from $\pi$ truncated to a computable intersection of intervals. The paper demonstrates this for non-conjugate posteriors (Poisson-log-normal, logistic regression, Weibull hazards), generalized linear mixed models, and non-linear mixed models, with a numerical comparison against Metropolis-Hastings (MH) showing sharply lower autocorrelation.

Key Claims

• Theorem 1: For $f(x) \propto \pi(x)\prod_i l_i(x)$ with each $l_i$ invertible, introducing $U_i \sim U(0, l_i(x))$ yields a Gibbs sampler whose conditionals are all uniform except one, which is $\pi$ restricted to $A_u = \{x: l_i(x) > u_i\ \forall i\}$.
• The auxiliary variable Gibbs sampler is equivalent to a Metropolis-Hastings independence chain with proposal $\pi$ but always moves — it never rejects, because sampling $U$ before $X$ conditions on the acceptance region.
• The decomposition $f = \pi\cdot\prod_i l_i$ is non-unique; this flexibility is exploited to make truncation sets analytically tractable (invertible $l_i$ gives closed-form $A_u$).
• For the logistic model and non-linear mixed models, dimension-by-dimension sampling (alternating through coordinates of $x$) handles multivariate $X$ when the joint set $A_u$ is intractable.
• Numerical example (orange-tree logistic growth, Lindstrom-Bates 1990): lag-1 autocorrelation 0.75 (auxiliary sampler) vs. 0.79 (MH random walk); MH autocorrelation still 0.39 at lag 40, auxiliary sampler reaches 0 by lag 9.
• The authors do not claim universal efficiency superiority; if an efficient MH or rejection algorithm exists, it may remain preferable.

Concepts Introduced or Extended

• Auxiliary Variable Sampler
• Data Augmentation
• Gibbs Sampler
• Markov Chain Monte Carlo

Entities Mentioned

• Paul Damien
• Jon Wakefield
• Stephen Walker
• Nicholas Polson
• Siddhartha Chib

Quotes

"We demonstrate that in many cases it is possible to introduce a latent variable so that all full conditionals are standard and can be sampled directly."

"The strength of this approach lies in its generality and its ease of implementation. The aim of the paper, therefore, is to provide an alternative sampling algorithm to rejection-based methods and other sampling approaches such as the Metropolis-Hastings algorithm."

My Take

The paper provides a clean, general theorem for non-conjugate Gibbs sampling that avoids the tuning overhead of MH. The "always moves" property is an elegant theoretical result. The main practical limitation is that finding an invertible decomposition $l_i$ still requires case-by-case ingenuity — the theorem guarantees existence of the Gibbs structure but not its tractability. The efficiency advantage in the orange-tree example is real but modest; for models where good MH proposals are available, the gain may not justify the extra coding. The connection to data augmentation (Tanner-Wong 1987) is acknowledged but the auxiliary variable framing is more explicit and generalizable.