Mixture Model

Definition

A finite mixture model represents the marginal density of an observable $y$ as a convex combination of $K$ component densities:

$$p(y|\theta) = \sum_{k=1}^K \pi_k \, f(y|\theta_k), \qquad \pi_k \geq 0, \quad \sum_{k=1}^K \pi_k = 1$$

The component densities $f(y|\theta_k)$ are typically from a parametric family (Normal, Poisson, etc.); the mixing weights $\pi_k$ and component parameters $\theta_k$ are unknown. Mixture models arise as models of population heterogeneity, as semiparametric approximations to arbitrary densities, and as data-generating processes in latent-class analysis.

Key Ideas

• Latent class interpretation: introducing an indicator variable $z_i \in \{1,\ldots,K\}$ such that $y_i | z_i = k \sim f(\cdot|\theta_k)$ makes the mixture a hierarchical model amenable to Expectation-Maximization (EM) and Gibbs sampling.
• Label switching: mixture posteriors are invariant to permutation of component labels — identifiability requires constraints (e.g., ordering means $\mu_1 < \mu_2 < \cdots$).
• Unbounded likelihood: without constraints on component variances, the Gaussian mixture likelihood is unbounded; regularisation via priors or minimum-variance constraints is needed.
• Number of components: $K$ can be fixed (finite mixture) or random (Dirichlet Process Mixture, which allows $K \to \infty$).

How It Works

EM algorithm: treats $z_i$ as missing data. E-step: compute $r_{ik} = \pi_k f(y_i|\theta_k) / \sum_j \pi_j f(y_i|\theta_j)$ (posterior class probabilities). M-step: update $\hat\pi_k = n^{-1}\sum_i r_{ik}$ and component parameters from weighted likelihoods.

Gibbs sampler: draw class indicators $z_i \sim \text{Categorical}(r_{i1},\ldots,r_{iK})$, then draw $(\pi_1,\ldots,\pi_K)$ from Dirichlet conjugate, and component parameters from conjugate posteriors conditional on class membership.

Why It Matters

Mixture models are the foundational tool for density estimation, clustering, and modelling population heterogeneity across econometrics, statistics, and machine learning.

Related

• Latent Class Model
• Dirichlet Process Mixture
• Mixture of Normals
• EM Algorithm
• Gibbs Sampler
• Carlin-Wolfe-Brown-Gelman (2001)