Mixture Model

Definition

A finite mixture model represents the marginal density of an observable $y$ as a convex combination of $K$ component densities:

$p(y|\theta) = \sum_{k=1}^K \pi_k \, f(y|\theta_k), \qquad \pi_k \geq 0, \quad \sum_{k=1}^K \pi_k = 1$

The component densities $f(y|\theta_k)$ are typically from a parametric family (Normal, Poisson, etc.); the mixing weights $\pi_k$ and component parameters $\theta_k$ are unknown. Mixture models arise as models of population heterogeneity, as semiparametric approximations to arbitrary densities, and as data-generating processes in latent-class analysis.

Key Ideas

Latent class interpretation: introducing an indicator variable $z_i \in \{1,\ldots,K\}$ such that $y_i | z_i = k \sim f(\cdot|\theta_k)$ makes the mixture a hierarchical model amenable to Expectation-Maximization (EM) and Gibbs sampling.
Label switching: mixture posteriors are invariant to permutation of component labels — identifiability requires constraints (e.g., ordering means $\mu_1 < \mu_2 < \cdots$ ).
Unbounded likelihood: without constraints on component variances, the Gaussian mixture likelihood is unbounded; regularisation via priors or minimum-variance constraints is needed.
Number of components: $K$ can be fixed (finite mixture) or random (Dirichlet Process Mixture, which allows $K \to \infty$ ).

How It Works

EM algorithm: treats $z_i$ as missing data. E-step: compute $r_{ik} = \pi_k f(y_i|\theta_k) / \sum_j \pi_j f(y_i|\theta_j)$ (posterior class probabilities). M-step: update $\hat\pi_k = n^{-1}\sum_i r_{ik}$ and component parameters from weighted likelihoods.

Gibbs sampler: draw class indicators $z_i \sim \text{Categorical}(r_{i1},\ldots,r_{iK})$ , then draw $(\pi_1,\ldots,\pi_K)$ from Dirichlet conjugate, and component parameters from conjugate posteriors conditional on class membership.

Why It Matters

Mixture models are the foundational tool for density estimation, clustering, and modelling population heterogeneity across econometrics, statistics, and machine learning.

Label Switching

The mixture likelihood is invariant under permutation of the component labels — for any permutation $\nu$ , $L(\nu(\theta))=L(\theta)$ (Redner–Walker 1984). With a symmetric prior, the posterior therefore has up to $k!$ symmetric modes, and the usual MCMC summaries (posterior means, marginal posteriors of component-specific parameters) average across components and become meaningless — the label-switching problem. Stephens (2000) shows that the common remedy of imposing an artificial identifiability constraint (e.g. ordering the means $\mu_1<\cdots<\mu_k$ ) fails in general, and instead proposes relabelling algorithms: treat the labelling as a decision and, for each MCMC draw, choose the permutation that minimizes a posterior expected loss defined on the induced clusterings of the data (a Kullback–Leibler loss on the classification probabilities). This is now the default approach, and the same symmetry afflicts Markov-switching and Dirichlet-process mixture models.

Mixture SEM

A structured-covariance variant restricts each component's covariance matrix $\Sigma_k$ by a postulated structural equation model (SEM) — a one-factor, latent-growth, or autoregressive structure — rather than leaving it unrestricted. This mixture SEM (latent class SEM; Vermunt-Magidson 2005) sits between the extremes of unrestricted MVN-mixture clustering (many covariance parameters) and the fully diagonal local-independence model, buying parsimony and stability while still capturing within-class dependence. Read the other way, it extends standard SEM to the case of unobserved group membership — a multiple-group analysis in which the groups are latent — so it detects unobserved heterogeneity such as class-specific error variances or growth patterns. Estimation is by ML or MAP via EM, with a mild prior (MAP) preventing the zero-variance boundary solutions that plague unrestricted mixtures.