Yang-Muthén-Yang (1999) Finite Mixture Multivariate Generalized Linear Models Using Gibbs Sampling and E-M Algorithms

Summary

A short methodological comparison of Expectation-Maximization (EM) and Gibbs sampling for estimating finite mixture multivariate generalised linear models. Proposes two new model extensions — an Extended Latent Class Analysis (ELCA) with covariate-predicted mixing probabilities, and a finite mixture multivariate Poisson model — and applies them to the Longitudinal Study of American Youth (LSAY) data. Implemented in Bayesian inference Using Gibbs Sampling (BUGS) with non-informative priors.

Key Claims

• Setting. Finite mixture GLMs assume the observed distribution arises from $C$ latent classes, each governed by its own GLM parameter vector; class membership is governed by mixing weights $\lambda_c = \Pr(C_i = c)$.
• Standard LCA (EM). Binary outcomes; $g_c(U_i) = \prod_k \pi_{kc}^{u_{ik}}(1-\pi_{kc})^{1-u_{ik}}$; posterior class probability $h_{ci} = \lambda_c g_c(U_i) / \sum_{c'}\lambda_{c'}g_{c'}(U_i)$. E-step replaces latent class $C_i$ with $h_{ci}$; M-step updates $\pi_{kc}$ and $\lambda_c$ by weighted maximum likelihood estimation (MLE). Iterate to convergence.
• New Model 1 — ELCA. Extends LCA by allowing covariates to predict conditional item probabilities ($\text{logit}(\pi_{ikc}) = \beta_{0kc} + \beta_{1kc}w_{1i} + \cdots$) AND mixing probabilities ($\text{logit}(\lambda_i) = \alpha_0 + \alpha_1 x_{1i} + \cdots$). Conjugate normal priors on all regression coefficients.
• New Model 2 — Finite Mixture Multivariate Poisson. Same two-level structure as ELCA but $y_{ik}|C_i \sim \text{Poisson}(\kappa_{ikc})$ with $\log(\kappa_{ikc}) = \beta_{0kc} + \beta_{1kc}w_{1i} + \cdots$; class membership predicted by covariates.
• EM vs Gibbs comparison (Monte Carlo). 20 replications, $n \in \{300, 800\}$, three starting-value sets per replication to avoid local modes. Gibbs sampling (BUGS, non-informative priors) yields slightly more accurate estimates and smaller standard deviations than EM. Both methods less stable at $n=100$.
• Computation time. Gibbs sampling required ~50 minutes (Pentium-100) for a 5k burn-in / 25k posterior run; EM converges in seconds.
• Real data — LSAY. 685 Grade 7 students, 5 binary math items. ELCA with three mixing-probability covariates (gender, mother's education, home science resources) and two conditional-probability predictors (math attitude and anxiety). Slopes constrained equal across items. Mother's education and home science resources significantly predict latent class membership; gender does not.

Concepts Introduced or Extended

• Latent Class Model

Entities Mentioned

• (none with existing wiki pages)

Quotes

"Our primary results show that Gibbs sampling gives more accurate estimates and smaller standard deviations than the E-M algorithm does although the differences are small in our examples."

"For the LCA model considered in this paper, we found that Gibbs sampling took much more time than the E-M algorithm did."

My Take

A minor technical note from a Taiwanese journal. The two proposed model extensions (covariate-predicted mixing weights in ELCA and mixture Poisson) are genuinely useful and were underexplored in Bayesian form at the time. The EM–Gibbs comparison is thin (20 replications, small Monte Carlo), and the conclusion that Gibbs is slightly more accurate with non-informative priors is expected — the two methods are theoretically equivalent in the limit. The practical contribution is confirmation that these models can be estimated in BUGS with reasonable sample sizes. The main lasting value is the educational application context and Muthén's involvement, which connects this to the broad mixture modelling literature in psychometrics. Not a core reference for time-series econometrics.