Auxiliary Variable Sampler

mcmcgibbs-samplerauxiliary-variablelatent-variablebayesiannon-conjugatetruncated-distributiondata-augmentation

Definition

The auxiliary variable sampler (Damien-Wakefield-Walker, DWW, 1999) is a Gibbs sampling technique for drawing from a target density f(x)∝π(x)∏i=1Nℓi(x)f(x) \propto \pi(x)\prod_{i=1}^N \ell_i(x) that cannot be sampled directly. For each factor ℓi\ell_i, a uniform latent variable Ui∼Uniform(0,ℓi(x))U_i \sim \text{Uniform}(0, \ell_i(x)) is introduced. The resulting Gibbs scheme cycles: (a) Ui∣x∼Uniform(0,ℓi(x))U_i \mid x \sim \text{Uniform}(0, \ell_i(x)) — trivial draw; (b) x∣u∼π(⋅)x \mid u \sim \pi(\cdot) restricted to the truncation set Au={x:ℓi(x)>ui,  ∀i}A_u = \{x: \ell_i(x) > u_i,\; \forall i\} — a draw from the prior truncated to a computable interval. The marginal distribution for xx is exactly ff.

Key Ideas

How It Works

Non-Conjugate Models

For f(x)∝ℓ(x)π(x)f(x) \propto \ell(x)\pi(x), introduce U∼Uniform(0,ℓ(x))U \sim \text{Uniform}(0, \ell(x)). The joint density p(x,u)∝π(x)1[u<ℓ(x)]p(x, u) \propto \pi(x) \mathbf{1}[u < \ell(x)] has marginal f(x)f(x).

Examples:

Model ℓ(x)\ell(x) π(x)\pi(x) Full conditional for x∣ux \mid u
Poisson–log-normal exp⁡(rx−ex)\exp(rx - e^x) N(0,1)\mathcal{N}(0,1) Truncated normal
Bernoulli–logistic ∏ihi(x)wibi(x)1−wi\prod_i h_i(x)^{w_i} b_i(x)^{1-w_i} N(0,1)\mathcal{N}(0,1) Truncated normal on interval AuA_u
Weibull hazards ∏iexp⁡(−uitiaeziβ)\prod_i \exp(-u_i t_i^a e^{z_i\beta}) N(μ,Σ)\mathcal{N}(\mu, \Sigma) Truncated normal

Generalized Linear Mixed Models (GLMM)

For the random-effects Poisson model yi∣θi∼Poisson(eθi)y_i \mid \theta_i \sim \text{Poisson}(e^{\theta_i}), θi=wiβ+bi\theta_i = w_i\beta + b_i:

  1. Introduce Ui∼Uniform(0,exp⁡(−yiθi))U_i \sim \text{Uniform}(0, \exp(-y_i \theta_i)) and Vi∼Exp(1)V_i \sim \text{Exp}(1) restricted to (eθi,∞)(e^{\theta_i}, \infty).
  2. Full conditional for each βk∣β−k,b,Λ,u,v\beta_k \mid \beta_{-k}, b, \Lambda, u, v is a truncated normal.
  3. Full conditional for bi∣β,Λ,u,vb_i \mid \beta, \Lambda, u, v is a truncated normal.
  4. Full conditional for precision Λ\Lambda is unchanged (Wishart or inverse-gamma).

Non-Linear Mixed Effects

For individual random effects θi\theta_i with non-log-concave conditional f(θi)∝∏jexp⁡(−12σ2[yij−g(θi)]2)⋅π(θi)f(\theta_i) \propto \prod_j \exp(-\frac{1}{2\sigma^2}[y_{ij} - g(\theta_i)]^2) \cdot \pi(\theta_i):

Introduce gamma latent effects uij∼Gamma(3/2,1/2)u_{ij} \sim \text{Gamma}(3/2, 1/2) restricted to (g(θi)2,∞)(g(\theta_i)^2, \infty). Each component θik∣θi,−k,u\theta_{ik} \mid \theta_{i,-k}, u becomes a draw from π(θik∣θi,−k)\pi(\theta_{ik} \mid \theta_{i,-k}) truncated to an analytically computable interval — no ratio-of-uniforms tuning required.

Numerical example (Lindstrom-Bates 1990 orange trees, n=7n=7, T=5T=5 obs each):

Algorithm Lag-1 autocorrelation (ACF) ACF at lag 40
Auxiliary Gibbs 0.75 ~0 (by lag 9)
Random-walk M-H 0.79 0.39

Why It Matters

Open Questions

Related