Imai and van Dyk (2005) A Bayesian Analysis of the Multinomial Probit Model Using Marginal Data Augmentation

Summary

Imai and van Dyk introduce new Markov chain Monte Carlo (MCMC) algorithms for Bayesian analysis of the multinomial probit model, using the framework of marginal data augmentation (Meng-van Dyk 1999). The identification constraint $\sigma_{11} = 1$ (scale normalization) makes direct Gibbs sampling awkward; existing algorithms handle it by either (a) placing a prior on the unidentifiable parameters, which produces an intractable implied prior on the identifiable parameters (McCulloch-Rossi 1994), (b) adding a Metropolis step to marginalize out the scale — Nobile (1998), who improves mixing but has an error in his acceptance probability — or (c) fixing the unidentifiable parameter and paying a severe convergence cost (McCulloch et al. 2000). The new algorithms place a prior directly on the identifiable $(\beta, \Sigma)$, use the unidentifiable scale $\alpha$ as a working parameter within the sampler (not fixing it, not putting an intractable prior on it), and involve only standard conjugate draws. Algorithm 1 (Scheme 1) dominates all existing methods on all three criteria: prior interpretability, computational speed, and simplicity.

Key Claims

• McCulloch-Rossi (1994) prior is uninterpretable: their prior $\tilde\beta \sim N(\tilde\beta_0, A^{-1})$ on the unidentified $\tilde\beta$ implies a marginal prior for the identifiable $\beta$ given by an intractable integral (eq. 16); it also cannot accommodate a flat prior on $\beta$.
• Nobile (1998) has an error: his Step 4 adds a Metropolis move intended to implement Scheme 1, but the acceptance probability $R$ substitutes $(\tilde\beta, \tilde\Sigma)$ for $(\beta, \Sigma)$ in error. The likelihood is not constant at $(\alpha, \tilde\beta, \tilde\Sigma)$ vs. $(\alpha', \tilde\beta, \tilde\Sigma)$. The correct $R$ (Appendix B) differs by a factor of $c^{-(2-k-p(p-1)/2)}$. Fig. 2 shows the practical effect is substantial — the uncorrected algorithm samples from the wrong distribution.
• McCulloch et al. (2000) is slow: fixing $\alpha$ eliminates the unidentifiable parameter from the chain, which removes the diffuseness of the augmented-data model and dramatically slows mixing (high autocorrelation, very sensitive to starting values).
• The key insight (Meng-van Dyk 1999): mixing improves when $p(W|\theta,Y)$ is as diffuse as possible. Marginal augmentation — averaging over a working prior for $\alpha$ — makes the augmented-data model $p(Y,W|\theta)$ more diffuse than conditional augmentation (fixing $\alpha = 1$), improving the geometric rate of convergence.
• Scheme 1 $\geq$ Scheme 2: marginalizing out the working parameter before the $W$-draw (Scheme 1) gives geometric rate of convergence no larger than conditioning on it (Scheme 2). McCulloch-Rossi (1994) uses Scheme 2; Nobile (1998) uses Scheme 1 — that is why it improves over McCulloch-Rossi, not primarily because of additional Metropolis mixing.
• New prior: $p(\Sigma) \propto |\Sigma|^{-(\nu+p)/2}\,[\text{trace}(S\Sigma^{-1})]^{-\nu(p-1)/2}$ subject to $\sigma_{11} = 1$; this is the distribution of $\tilde\Sigma/\tilde\sigma_{11}$ where $\tilde\Sigma \sim \text{inv-Wishart}(\nu, \tilde S)$. The working prior is $\alpha^2 \mid \Sigma \sim \alpha^2_0\,\text{trace}(S\Sigma^{-1})/\chi^2_{\nu(p-1)}$.
• Algorithm 1 (Scheme 1) dominates: as fast as Nobile (corrected), better prior than McCulloch-Rossi, no Metropolis step needed, flat priors on $\beta$ allowed. Detergent data: converges in ~1000 draws; Nobile/McCulloch-Rossi needed 60,000–80,000 draws from a bad starting value.

Concepts Introduced or Extended

• Multinomial Probit — marginal data augmentation algorithm; new prior on identifiable parameters; Nobile (1998) error correction
• Data Augmentation — marginal vs. conditional augmentation; working parameter framework (Meng-van Dyk 1999)
• Metropolis-Hastings Algorithm — Nobile's corrected acceptance probability

Entities Mentioned

• Kosuke Imai
• David A. van Dyk
• Robert E. McCulloch
• Chib-Greenberg (1998)

Quotes

"When comparing statistical methods, computational performance should nearly always be a less important criterion than the choice of model specification."

"It would seem that sometimes there is such a thing as a free lunch!"

My Take

The paper makes three distinct contributions: (1) the new prior on identifiable parameters — genuinely useful in practice; (2) the theoretical explanation of why Nobile's algorithm works (it's Scheme 1) and the correction of his error — important for reproducibility; (3) the empirical demonstration that the combination dominates all prior methods. The unification of McCulloch-Rossi, Nobile, and McCulloch et al. under the conditional/marginal augmentation framework is also didactically valuable. The comparison to Edwards-Allenby (2003) is instructive: both handle non-identified probit likelihoods via working parameters, but Imai-van Dyk explicitly manage the working parameter within the chain iteration while Edwards-Allenby postprocess draws after the fact.