Holmes-Held (2006) Bayesian Auxiliary Variable Models for Binary and Multinomial Regression

Summary

Holmes and Held extend the Albert-Chib (1993) probit auxiliary variable Gibbs sampler in three directions: (1) a joint update of $(\beta, z)$ in the probit model that substantially reduces autocorrelation; (2) an exact auxiliary variable sampler for logistic regression via the Kolmogorov-Smirnov scale mixture of normals representation — in contrast to Albert-Chib's approximate $t_8$ trick; (3) direct extension to polychotomous (multinomial) logistic regression by conditioning on other class coefficients. All algorithms are fully automatic with no user-set tuning parameters or necessary Metropolis-Hastings (MH) accept/reject steps.

Key Claims

• Joint probit update: Updating $\pi(\beta, z \mid y) = \pi(z \mid y)\pi(\beta \mid z)$ instead of the separate A&C schedule reduces posterior correlation; leave-one-out marginal predictive updates give $v_i = 1 + w_i > 1$ (inflated vs. standard $v_i=1$); ~2× improvement in effective sample size (ESS), ~2× larger update distance, at near-zero extra computation.
• Exact logistic sampler: $\varepsilon_i \sim \text{Logistic}(0,1)$ written as $N(0, \lambda_i)$ with $\lambda_i = (2\psi_i)^2$, $\psi_i \overset{\text{iid}}{\sim} \text{KS}$ (Kolmogorov-Smirnov). This gives exact logistic likelihood. $\beta \mid z, \lambda \sim N(B, V)$ with $V = (v^{-1} + x'Wx)^{-1}$, $W = \text{diag}(\lambda_i^{-1})$ — conjugate update at each $\lambda$ draw.
• Two logistic joint-update schemes: Scheme A ($\{z, \lambda\}$ jointly: truncated logistic sampling by inversion, $z_i \mid \beta, y_i \sim \text{Logistic}(x_i\beta, 1)$ truncated; then $\lambda \mid z, \beta$ by rejection with generalized inverse Gaussian (GIG) envelope) — recommended for simplicity and vectorisation. Scheme B ($\{z, \beta\}$ jointly given $\lambda$) — larger jumps but more autocorrelated. Scheme A wins on ESS; Scheme B on mean step distance.
• Variable selection (Section 2.5): Condition on $\{z, \lambda\}$; update $(\gamma, \beta)$ jointly. MH acceptance = Bayes factor over model space: $\alpha = \min\!\left(1, \frac{|V_{\gamma^*}|^{1/2}|v_\gamma|^{1/2}}{|V_\gamma|^{1/2}|v_{\gamma^*}|^{1/2}} \frac{\exp(0.5 B_{\gamma^*}'V_{\gamma^*}^{-1}B_{\gamma^*})}{\exp(0.5 B_\gamma' V_\gamma^{-1} B_\gamma)} \frac{\pi(\gamma^*)q(\gamma\mid\gamma^*)}{\pi(\gamma)q(\gamma^*\mid\gamma)}\right)$. Draws of $\beta$ do not appear — marginalisation gives efficient dimension jumps. Pima Indian application: acceptance ~4%; Gl (glucose) and BMI (body mass index) $>99\%$ inclusion, BP (blood pressure) and TST (triceps skinfold thickness) $<4\%$.
• Polychotomous (Section 3): For $Q$ classes, condition on $\beta_{-j}$ → conditional likelihood for $\beta_j$ has logistic form on indicator $\mathbf{1}[y_i = j]$ with offset $C_{ij} = \log\sum_{k\neq j}\exp(x_i\beta_k)$. Loop Scheme A over $j = 1,\ldots,Q-1$.
• Rao-Blackwellization (Section 3): Marginal posterior $\hat{p}(\beta^*\mid y) = M^{-1}\sum_j N(B^{*(j)}, V^{*(j)})$ — a Gaussian mixture with known normalizing constant, enabling direct density and credible-region estimation.
• Comparison: Albert-Chib's $\lambda_i \sim \text{Gamma}(4,4)$ giving $t_8$ approximation departs from logistic in the tails (Figure 2 qq-plot). Gamerman (1997) weighted least squares (WLS)-MH proposal works but requires accept/reject and is non-trivial for variable-dimension settings.

Concepts Introduced or Extended

• Bayesian Logistic Regression
• Data Augmentation
• Binary Probit
• Auxiliary Variable Sampler
• Variable Selection

Entities Mentioned

• Chris C. Holmes
• Leonhard Held
• Siddhartha Chib
• James H. Albert

Quotes

"All of the algorithms are fully automatic with no user set parameters and no necessary Metropolis-Hastings accept/reject steps."

"Our approach is exact, fully automatic (no accept-reject) and we present extensions to multinomial (multi-class, polychotomous) regression."

My Take

The paper's central contribution — replacing the approximate $t_8$ scale mixture with the exact KS-logistic mixture — is clean and consequential: it removes a source of tail bias that is hard to diagnose in practice. The joint probit update is a straightforward but high-value tweak to A&C with essentially no downside. The variable-selection extension is elegant: by conditioning on $\{z,\lambda\}$ the dimension-jump acceptance collapses to a Bayes factor form, avoiding the usual instability of reversible jump MCMC (RJMCMC) in regression settings. The Rao-Blackwellization insight (known normalising constant in auxiliary variable settings) is underappreciated and directly usable for credible-region computation in high-dimensional binary regressions.