Bayesian Residual Analysis for Binary Response Regression Models

Summary

Classical residuals for binary regression — Pearson $(y_i - \hat{p}_i)/\{\hat{p}_i(1-\hat{p}_i)\}^{1/2}$ and deviance — have unknown sampling distributions, making outlier plots hard to calibrate. Albert and Chib (1995) define two Bayesian alternatives with interpretable posterior distributions and show that both emerge naturally from the Albert-Chib (1993b) Gibbs sampler at no additional computational cost. The paper demonstrates both residual types on simulated data and two real datasets (Brown 1980 prostate cancer; Kenward-Jones 1987 crossover trial), and extends the latent residual approach to longitudinal random effects probit models.

Key Claims

Parametric residual $r_i = y_i - p_i$ where $p_i = F(x_i^T\beta)$ : bounded on $(y_i-1, y_i)$ , has a continuous posterior despite discrete $y_i$ . Outlier signal is a posterior concentrated near the endpoint of its support interval. Outlying probability $\text{pr}(|r_i| > K|y)$ evaluated at $K = 0.75$ .
Latent data residual (probit) $\varepsilon_i = Z_i - x_i^T\beta$ : a priori $\mathcal{N}(0,1)$ , providing a calibrated reference. Posterior (eq. 3) is a truncated normal: $\pi(\varepsilon_i|y_i,\beta) \propto \phi(\varepsilon_i)\cdot I(\varepsilon_i > -x_i^T\beta)$ if $y_i=1$ , and $\propto \phi(\varepsilon_i)\cdot I(\varepsilon_i < -x_i^T\beta)$ if $y_i=0$ . Outlying probability $\text{pr}(|\varepsilon_i| > K|y)$ compared against prior tail $2[1-\Phi(K)]$ .
Free from the Gibbs run: latent residuals are by-products of the Albert-Chib (1993b) two-block Gibbs sampler — record $\varepsilon_i^{(g)} = Z_i^{(g)} - x_i^T\beta^{(g)}$ at each iteration at no extra cost.
Rao-Blackwellised estimates: marginal posterior density $\pi(\varepsilon_i|y) = \int \pi(\varepsilon_i|y,\beta)\pi(\beta|y)\,d\beta$ estimated by averaging the conditional truncated-normal density over Gibbs draws (Gelfand-Smith 1990), giving a smoother estimate than raw simulation quantiles.
Extensions: logistic link — cannot simulate $Z_i$ directly; Rao-Blackwellise over Markov chain Monte Carlo (MCMC) draws of $\beta$ (Dellaportas-Smith 1993). Longitudinal random effects probit (eq. 4): $\text{pr}(y_{it}=1|b_i) = \Phi(x_{it}^T\beta + b_i)$ , latent residual $\varepsilon_{it} = Z_{it} - x_{it}^T\beta$ has prior $\mathcal{N}(0, 1+\sigma^2)$ ; outlying criterion $\text{pr}(|\varepsilon_{it}| > K(1+\sigma^2)^{1/2}|y)$ (eq. 5).
Real data: Brown (1980) prostate cancer ( $N=53$ ) — badly fitting model (age + grade) shows no outliers; well-fitting model (log-acid + X-ray + size + grade) flags obs 9, 26, 35, 37 with $\text{pr}(|\varepsilon_i|>2|y) \in \{0.30, 0.23, 0.20, 0.30\}$ . Kenward-Jones (1987) crossover trial ( $N=86$ , 3 periods): random effects probit identifies several anomalous response patterns.

Concepts Introduced or Extended

Entities Mentioned

Quotes

"In contrast, Bayesian residuals have continuous-valued posterior distributions which can be graphed to learn about outlying observations." (Summary)

"These latent residuals are particularly interesting to study since they can be simulated and summarised as a by-product of the algorithm of Albert & Chib (1993)." (p. 750)

My Take

The paper's key insight is elegantly simple: the Albert-Chib (1993b) data augmentation already produces $Z_i^{(g)}$ and $\beta^{(g)}$ at each Gibbs iteration, so the latent residual $\varepsilon_i^{(g)} = Z_i^{(g)} - x_i^T\beta^{(g)}$ is available for free. The $\mathcal{N}(0,1)$ prior reference for the probit case makes outlier detection substantially more interpretable than classical Pearson/deviance residuals, which have unknown sampling distributions. The Rao-Blackwellised density estimator is a useful addition over raw quantiles. The extension to longitudinal random effects probit (eq. 5) is clean, though the paper stops short of a full simulation study for that case. The approach is less elegant under the logistic link, where $Z_i$ cannot be simulated directly and only Rao-Blackwellised densities are available.