Antoniak (1974) Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems

Summary

Antoniak extends Ferguson's (1973) Dirichlet process (DP) — a random probability measure whose sample paths are almost surely distributions, used as a prior over the space of distributions in Bayesian nonparametrics — to the case where the random measure is a mixing distribution for a parameter that governs the sampling distribution. In that setting the posterior for the random measure is no longer a single Dirichlet process but a mixture of Dirichlet processes (MDP). The paper defines MDPs formally, proves several properties (most importantly a closure property under conditioning on data), derives formulas for the posterior, and works applications in bio-assay, discrimination, regression, and estimation of a mixing distribution.

Key Claims

• Mixture of Dirichlet processes (MDP). If the DP's parameter (its base measure, or transition/concentration structure) is itself random with some mixing distribution, the resulting prior is a mixture of Dirichlet processes; Ferguson's single-DP posterior is the special case with a degenerate mixing distribution.
• Closure under sampling. The class of MDPs is closed under posterior updating: conditioning an MDP prior on observations yields another MDP (with an updated mixing distribution). This is the central theorem — it makes iterated Bayesian nonparametric inference tractable.
• Distribution of the number of distinct values (the "Antoniak equation"). For $n$ observations drawn (via the Blackwell–MacQueen Pólya-urn scheme) from a DP with concentration/total mass $\alpha = M$, the number $k$ of distinct values (clusters) has $$P(k=j\mid n) = c_n(j)\,n!\,M^{j}\,\frac{\Gamma(M)}{\Gamma(M+n)},$$ where $c_n(j)$ are (unsigned Stirling) coefficients, giving $E[k]\approx M\log\!\left(1+\tfrac{n}{M}\right)$. The number of clusters therefore grows only logarithmically in $n$ and is controlled by $M$.
• Learning the concentration parameter. Because $k$ is informative about $M$, the number of observed distinct values provides the likelihood for placing a prior on (and updating) $M$ — the basis of later concentration-parameter samplers (Escobar–West).
• Pólya-urn / Blackwell–MacQueen representation. The predictive rule "reuse an existing value with probability proportional to its count, or draw a fresh value from the base measure with probability proportional to $M$" underlies both the clustering behavior and the computational schemes.
• Applications. Bio-assay (monotone dose–response with a DP prior on the response distribution), discrimination/classification, nonparametric regression, and estimation of an unknown mixing distribution.

Concepts Introduced or Extended

• Dirichlet Process Mixture
• Mixture Model

Entities Mentioned

• Charles E. Antoniak
• Thomas S. Ferguson

Quotes

"This paper extends Ferguson's result to cases where the random measure is a mixing distribution for a parameter which determines the distribution from which observations are made."

"The conditional distribution of the random measure, given the observations, is no longer that of a simple Dirichlet process, but can be described as being a mixture of Dirichlet processes."

My Take

This paper is the theoretical hinge between Ferguson's (1973) Dirichlet process and the modern Dirichlet process mixture models that dominate Bayesian nonparametric clustering and density estimation. Two of its results do the heavy lifting downstream: the MDP closure property, which guarantees the posterior stays in a manageable class, and the distribution of the number of distinct clusters, which is exactly what lets practitioners reason about — and place a prior on — the concentration parameter $M$ that governs how many mixture components the data will support. Escobar–West's concentration-parameter sampler, MacEachern–Müller's and Neal's (2000) DP-mixture MCMC, and every "infinite mixture" application trace their tractability to the machinery Antoniak set down here. The paper is heavily measure-theoretic, but its practical residue — the Pólya-urn predictive rule and the $E[k]\approx M\log(1+n/M)$ scaling — is what working Bayesians actually use.