Multivariate Probit

multivariate-probitprobitlatent-variablefactor-analysismaximum-likelihoodem-algorithmmarginal-likelihoodbayesiangibbs-samplersparse-factor-probitpick-any-jidentificationcorrelation-matrix

Definition

The multivariate probit model jointly analyzes pp binary outcome variables for the same subject by positing an underlying pp-dimensional continuous latent response vector yi\mathbf{y}_i distributed multivariate-normally. Subject ii is observed to succeed on outcome jj (uij=1u_{ij}=1) whenever the jj-th latent variable exceeds zero. Correlation among binary outcomes is captured via shared latent factors rather than a free p×pp\times p correlation matrix, making the model tractable for p>2p>2.

Key Ideas

How It Works

Estimation (Bock-Gibbons 1996; Gibbons-Wilcox-Gok 1998):

  1. Expectation-Maximization (EM) initialization: Run EM cycles starting from λjk=1k=j\lambda_{jk}=\mathbf{1}_{k=j} and dj=1d_j=1 to obtain suitable Λ\Lambda values.
  2. Fisher scoring: Fix m(m+1)/2m(m+1)/2 elements of Λ\Lambda in lower-triangular form to resolve rotational indeterminacy; score the remaining parameters with standard errors from the information matrix (Aitchison-Silvey constrained approach or direct fixing).
  3. Quadrature: Gauss-Hermite with Q≥5Q \geq 5 points per dimension approximates the mm-fold integral over θ\boldsymbol{\theta}.
  4. Empirical prior (Bock-Aitkin 1981): Replace Gaussian quadrature weights with nonparametric posterior-density estimates g(Tk)∝∑iP(ui∣Tk)A(Tk)g(\mathbf{T}_k) \propto \sum_i P(\mathbf{u}_i|\mathbf{T}_k)A(\mathbf{T}_k); iterate to joint convergence.
  5. Sampling weights: log⁡L=∑irilog⁡P(ui)\log L = \sum_i r_i \log P(\mathbf{u}_i) where rir_i are observation weights; derivatives scale accordingly.

Model selection: Change in log-likelihood (deviance) as factors are added; chi-squared approximation satisfactory for deviance differences even with ungrouped data. Absolute fit testable only when covariates are excluded (response patterns can be grouped into 2p2^p cells).

Identification: pq+pm−m(m+1)/2pq + pm - m(m+1)/2 estimable parameters — equals the saturated model mean parameters plus pairwise correlations. Lower-triangular Λ\Lambda (set λjk=0\lambda_{jk}=0 for k>jk>j) removes rotational indeterminacy.

Why It Matters

Sparse Factor Analytic Probit Extension (Hahn-Carvalho-Scott 2012)

For high-dimensional binary outcomes (pp large relative to nn), Hahn, Carvalho, and Scott (2012) replace the unstructured Σ\Sigma with a low-rank sparse decomposition: Σ=BVB⊤+I\Sigma = BVB^\top + I, where BB is a p×kp \times k lower-triangular loading matrix with spike-and-slab priors on each entry, and VV is a free k×kk \times k positive-definite factor covariance.

Key extensions relative to Chib-Greenberg (1998):

See Sparse Factor Probit and Hahn, Carvalho, Scott (2012).

MCMC for High-Dimensional Pick Any/J Data (Edwards-Allenby 2003)

"Pick any/JJ" surveys — where respondents check all that apply from JJ items — yield multivariate binomial responses. The identification challenge is that the JJ-dimensional latent normal must have unit variances (i.e., a correlation matrix RR, not a covariance matrix Σ\Sigma), but directly sampling RR in Markov Chain Monte Carlo (MCMC) is computationally hard for J>4J > 4.

Edwards and Allenby (2003) resolve this by ignoring the restriction during sampling:

  1. Fit an unrestricted covariance matrix Ω\Omega under an inverted-Wishart prior; draw (yk,Ω,γ)(\mathbf{y}_k, \Omega, \gamma) from the standard three-block Gibbs sampler.
  2. Postprocess each draw: let C=diag(Ω)−1/2C = \mathrm{diag}(\Omega)^{-1/2}, then α^=Cγ\hat\alpha = C\gamma and R^=CΩC′\hat R = C\Omega C'.

Key results: (i) R^\hat R is guaranteed positive definite; (ii) mean squared error (MSE) is lower than tetrachoric correlations at all tested sample sizes (Tables 1–3); (iii) tetrachoric correlations fail entirely when any cell count is zero, while the Bayesian estimator is always identified through the prior. The method handles J≥50J \geq 50 easily and unlocks standard multivariate analysis (principal component analysis (PCA), canonical correlation, conditional normals) on the estimated latent correlation matrix.

The general principle: For any non-identified likelihood with a proper prior, the posterior concentrates on the identified subspace. MCMC chains on individual non-identified parameters wander indefinitely, but postprocessed functions of those draws that are identified converge normally. Identification shifts from a binary (likelihood identified or not) to a continuous concern about the degree of posterior concentration.

See Edwards and Allenby (2003).

Open Questions

Related