Albert-Chib (1993b) Bayesian Analysis of Binary and Polychotomous Response Data

Summary

Albert and Chib (1993b) show that binary and polychotomous probit models become analytically tractable under data augmentation: introducing latent continuous responses $Z_i$ converts the truncated-likelihood problem into a two-block Gibbs sampler with Normal and truncated-Normal full conditionals. Extensions cover t-link scale mixtures (robust to outliers), hierarchical priors (enabling variable selection), ordered multinomial outcomes (adding a threshold-sampling block), and unordered multinomial probit (multivariate truncated Normal draws). All cases are unified by the data augmentation principle and validated on three classic datasets.

Key Claims

• Binary probit Gibbs (eqs. 5–6): Introduce latent $Z_i \sim \mathcal{N}(x_i'\beta, 1)$ with $Y_i = \mathbf{1}[Z_i > 0]$. Full conditionals: $\beta|Z \sim \mathcal{N}_k((X'X)^{-1}X'Z,\,(X'X)^{-1})$ and $Z_i|y_i,\beta \sim \mathcal{N}(x_i'\beta, 1)$ truncated to $(0,\infty)$ if $y_i = 1$ and to $(-\infty, 0]$ if $y_i = 0$. Gibbs converges to exact posterior at 800 iterations on Finney (1947) toxicity data (39 obs, 3 covariates).
• Robust t-link (eqs. 7–11): Scale mixture $Z_i|\lambda_i \sim \mathcal{N}(x_i'\beta, \lambda_i^{-1})$, $\lambda_i \sim \text{Gamma}(\nu/2, 2/\nu)$, $\nu$ discrete on $\{4, 8, 16, 32\}$. Four-block Gibbs; logistic link well-approximated by $t(8)$. Applied to 1976 Carter–Ford election data where maximum likelihood estimation (MLE) fails for the South-East regional variable; t-link and probit give qualitatively similar results.
• Hierarchical prior (eqs. 13–14): $\beta \sim \mathcal{N}(A\beta^0, \sigma^2 I)$; posterior $\beta|Z,\sigma^2$ Normal with augmented design matrix; $\sigma^2|Z$ non-standard but grid-sampable. Enables regional shrinkage to zero in the election example.
• Ordered multinomial (eqs. 16–18): $Y_i = j$ iff $\gamma_{j-1} < Z_i \leq \gamma_j$; three-block Gibbs adds threshold block $\gamma_j|Z,\beta,y \sim \text{Uniform}[\max\{Z_i : Y_i = j\},\, \min\{Z_i : Y_i = j+1\}]$.
• Unordered multinomial probit (eqs. 19–21): $\mathbf{Z}_i \sim \mathcal{N}_J(X_i\beta, \Sigma)$; $Y_i = j$ iff $Z_{ij}$ is maximum. Multivariate-truncated-Normal draw via accept-reject; Daganzo trivariate probit (50 obs): posterior mean $\hat\rho = 0.234$ vs. MLE 0.238.

Concepts Introduced or Extended

• Binary Probit
• Data Augmentation
• Multinomial Probit
• Bayesian Residual Analysis
• Gibbs Sampler

Entities Mentioned

• James H. Albert
• Siddhartha Chib

Quotes

"The basic idea of this article is that by using the method of data augmentation (Tanner and Wong 1987), the regression coefficients can be updated by a simple standard step." (p. 669)

"One advantage of the Bayesian approach is that the information from the latent variable $Z_i$ can be used to form residuals $r_i = y_i - p_i$..." (p. 674)

My Take

The paper's enduring contribution is demonstrating that data augmentation completely dissolves the computational difficulty of binary regression. The Gibbs sampler is exact (up to Markov chain Monte Carlo (MCMC) error) where classical MLE is unreliable or breaks down entirely (election SE variable). The hierarchical and t-link extensions foreshadow modern robust Bayesian workflows. A subtle point: the latent variable formulation makes Bayesian residuals $y_i - \Phi(x_i'\beta)$ real-valued and continuous — a diagnostic tool unavailable to classical probit. The multinomial extensions are more heuristic (accept-reject for multivariate truncated Normal is slow for large $J$), and later work (McCulloch-Rossi 1994) improved on this.