Logistic Variable Selection via Pólya–Gamma SSVS

Python · R · Part 9 of the variable-selection arc — binary outcomes

Overview

Part 9 of the variable-selection arc: selection for a binary outcome. Most applied selection problems have a binary response — disease or not, default or not — which breaks the clean conjugacy the earlier notebooks relied on: the logistic likelihood is not Gaussian, so the spike-and-slab Gibbs of Part 1 does not apply directly. The elegant fix is Pólya–Gamma data augmentation (Polson, Scott & Windle 2013): introduce a latent weight ωi\omega_i per observation and the logistic likelihood becomes conditionally Gaussian in the coefficients — after which the exact same conjugate SSVS sweep works. The showcase is the South African Heart Disease data (the logistic example in Elements of Statistical Learning): 462 men, coronary heart disease against nine risk factors — which of the nine belong in the model? Because p=9p=9 is small, we run the SSVS search and enumerate all 29=5122^9=512 logistic models exactly — bridging Part 1 and Part 2 for a binary outcome — cross-checked against PyMC and R's BAS.

ωi∣β∼PG(1, xi′β)  ⟹  β∣ω,γ∼N (weighted LS),zi=yi−1/2ωi\omega_i\mid\beta \sim \text{PG}(1,\ x_i'\beta) \;\Longrightarrow\; \beta\mid\omega,\gamma \sim \mathcal N \ \text{(weighted LS)}, \quad z_i = \tfrac{y_i-1/2}{\omega_i}

The Pólya–Gamma trick — a non-conjugate likelihood made Gaussian

Logistic regression with a spike-and-slab prior on the standardized predictors: Pr⁡(yi=1)=1/(1+e−xi′β)\Pr(y_i=1)=1/(1+e^{-x_i'\beta}) with the usual narrow-spike / wide-slab prior and γj∼Bernoulli(w)\gamma_j\sim\text{Bernoulli}(w). The likelihood has no conjugate prior, so β∣data\beta\mid\text{data} is not a tidy normal and the Part-1 Gibbs stalls. Polson–Scott–Windle show that with a latent Pólya–Gamma variable ωi∼PG(1,xi′β)\omega_i\sim\text{PG}(1, x_i'\beta) per observation, the likelihood contribution becomes Gaussian in β\beta with pseudo-response (yi−12)/ωi(y_i-\tfrac12)/\omega_i and weight ωi\omega_i. So the Gibbs sweep is: (1) draw ωi∣β∼PG(1,xi′β)\omega_i\mid\beta\sim\text{PG}(1, x_i'\beta) with the Devroye/PSW exact sampler; (2) draw β∣ω,γ\beta\mid\omega,\gamma as a weighted-least-squares normal with the spike-and-slab prior variances, exactly as in Part 1; (3) draw the indicators γj\gamma_j from the spike/slab density ratio. And exactly: with only nine predictors we also enumerate all 292^9 models, each logistic model's marginal likelihood by a Laplace approximation — the logistic likelihood has no conjugate prior, so the integral over β\beta has no closed form, and Laplace supplies one by fitting a Gaussian at the posterior mode, whose curvature there is the binomial Fisher information X′WXX'WX. Because the log-posterior of a logistic regression is concave and n=462n=462 is comfortably larger than nine predictors, that Gaussian is a close fit and the resulting model and inclusion probabilities are exact for practical purposes. Engine: a self-contained logitsel.py.

Results

Five of nine risk factors survive. Both the Pólya–Gamma search and the exact enumeration keep age, family history, tobacco, type-A behaviour and LDL cholesterol, and drop systolic BP, adiposity, obesity and alcohol — exactly the model Elements of Statistical Learning reaches by stepwise selection, recovered here as a posterior over models with uncertainty attached. age is certain (inclusion ≈ 1), family history nearly so, the others firmly in but graded. A famous non-finding: selection is about conditional, not marginal, relevance — and here the two genuinely part ways. Adiposity, on its own about as correlated with heart disease as the retained LDL (r≈0.25r\approx0.25), is dropped because its signal is already carried by the factors it travels with (age, obesity, LDL); type-A behaviour, only weakly correlated marginally (r≈0.10r\approx0.10), is kept for the independent signal it adds — the logistic posterior sees through the confounding. The selected model is also interpretable: coronary risk climbs ~5% in odds per year of age, and each retained factor (family history most) shifts the whole age–risk curve up additively on the log-odds scale, tracking the observed proportions. Selection is free here: the five-variable model's predicted probabilities track the full nine-variable model's almost exactly (r=0.985r=0.985) and generalize slightly better out of sample (CV AUC 0.782 vs 0.770) — the dropped factors were contributing variance, not signal. Three ways agree: the from-scratch Gibbs, PyMC's native logistic SSVS, and exact enumeration all select the same five — the augmentation is what lets the whole Gaussian selection machinery of Part 1 carry over to binary data (and, unchanged, to counts and multinomial beyond).

Notebooks

Downloads

References