Sparse Factor Probit

Definition

The sparse factor analytic probit model is a Bayesian multivariate probit that replaces an unstructured covariance matrix with a low-rank factor decomposition $\Sigma = BVB^\top + I$ , where the $p \times k$ loading matrix $B$ is sparse via spike-and-slab priors on each entry. It extends Chib-Greenberg (1998) multivariate probit to high-dimensional binary outcomes by combining Geweke-Zhou (1996) triangular identification, free correlated factor covariance $V$ , and automatic multiplicity correction via $\text{Beta}(1,1)$ inclusion probabilities.

Key Ideas

For each of $n$ observations: $z_i \sim \mathcal{N}(\alpha + \lambda\gamma_i + Bf_i, I_p)$ ; $y_{ij} = \mathbf{1}[z_{ij} > 0]$
$\gamma_i$ is a scalar "partisanship" factor (or shared latent trait); $\lambda$ is a $p$ -vector of loadings on $\gamma_i$
$B$ is a $p \times k$ lower triangular matrix of factor loadings with $b_{jj} > 0$ ; $b_{js} = 0$ for $s > j$ (Geweke-Zhou 1996 identification)
$f_i \sim \mathcal{N}(0, V)$ ; $V$ is a free $k \times k$ positive-definite matrix (not constrained to $I_k$ )
Spike-and-slab on each $b_{js}$ : $(b_{js} \mid \nu_s, q_s) \sim q_s \cdot \mathcal{N}(0, \nu_s) + (1-q_s) \cdot \delta_0(b_{js})$
$\nu_s \sim \text{IG}$ (inverse-gamma (IG), slab variance); $q_s \sim \text{Beta}(1,1)$ (inclusion probability per factor column)
Posterior inclusion: $\hat{q}_{js} = \hat{r}_{js} / [(1-q_s)/q_s + \hat{r}_{js}]$ where $\hat{r}_{js}$ is the Bayes factor ratio

How It Works

The 12-step Gibbs sampler cycles over:

Intercepts $\alpha$ (regression)
Partisanship loadings $\lambda$
Partisanship scores $\gamma_i$ (truncated normal by group membership)
Partisanship hyperparameters
Factor scores: $f_i \sim \mathcal{N}(m, M)$ with $M = (V^{-1} + B^\top B)^{-1}$ , $m = M \cdot B^\top(z_i - \alpha - \lambda\gamma_i)$
Factor covariance: $V \sim \text{IW}(V_0 + FF^\top, n + h)$ (inverse-Wishart (IW))
Spike-and-slab loading update per (j, s): compute Bayes factor $\hat r_{js}$ , draw $b_{js}$ from mixture 8–9. Slab variance ν_s and inclusion probability q_s updates
Latent utilities z_i (truncated normal Albert-Chib data augmentation) 11–12. Additional covariate regression parameters

Free factor covariance $V$ fosters sparsity in $B$ : correlation across factors is absorbed by off-diagonal entries of $V$ , so fewer non-zero loadings in $B$ are needed compared to orthogonal-factor ( $V = I$ ) models. If $\tilde{B} = BL$ is the orthogonal rotation, $\tilde{B}$ is generally denser than $B$ .

Why It Matters

Unstructured covariance estimation fails in high dimensions (p large relative to n): the Wishart estimator is severely biased and the number of parameters grows as $p(p+1)/2$ . A $k$ -factor structure reduces this to $O(pk)$ parameters, and sparsity further reduces effective dimensionality. The $\text{Beta}(1,1)$ multiplicity correction (Scott-Berger 2006, 2010) automatically adjusts for the large number of loading tests, avoiding spurious inclusions without a separate Bonferroni correction. In simulation at $p=100$ , $n=50$ , the sparse-6-factor model achieves Stein loss of 43.4 versus 503.1 for the Wishart benchmark.

Open Questions