Ferguson (1973) A Bayesian Analysis of Some Nonparametric Problems

Summary

Ferguson introduces the Dirichlet process (DP) as a prior distribution over the space of probability measures, launching Bayesian nonparametrics. He identifies two normally antagonistic desiderata for a nonparametric prior — large support (I) and an analytically manageable posterior (II) — and shows the Dirichlet process achieves both. The paper defines the DP via finite-dimensional Dirichlet distributions, proves its conjugacy under i.i.d. sampling, shows the realizations are discrete with probability one, and applies it to several nonparametric problems.

Key Claims

• Definition. For a finite non-null base measure $\alpha=MG_0$ on the sample space, $G\sim\text{DP}(\alpha)$ means that for every finite measurable partition $(B_1,\ldots,B_k)$, $(G(B_1),\ldots,G(B_k))\sim\text{Dirichlet}(\alpha(B_1),\ldots,\alpha(B_k))$; hence $E[G(B)]=G_0(B)$ and $M=\alpha(\text{whole space})$ is the concentration/total-mass parameter.
• Conjugacy (main theorem). If $G\sim\text{DP}(\alpha)$ and $X_1,\ldots,X_n$ is a sample from $G$, the posterior is again a Dirichlet process: $G\mid X_{1:n}\sim\text{DP}\!\big(\alpha+\sum_{i=1}^n\delta_{X_i}\big)$, where $\delta_x$ is a point mass at $x$.
• Resolving the antagonism. The DP simultaneously has large support (property I) and a tractable, conjugate posterior (property II) — the combination earlier priors could not achieve.
• Discreteness. An alternative constructive definition (Section 4) shows the DP assigns probability one to the set of discrete probability measures — contrasting with methods (Dubins-Freedman) that yield singular continuous distributions.
• Applications. Estimation of a distribution function, of its mean, of quantiles, and of related functionals, plus two-sample and other classical nonparametric problems, are carried out in closed form under the DP prior.

Concepts Introduced or Extended

• Dirichlet Process Mixture

Entities Mentioned

• Thomas S. Ferguson

Quotes

"This paper presents a class of prior distributions, called Dirichlet process priors, broad in the sense of (I), for which (II) is realized, and for which treatment of many nonparametric statistical problems may be carried out."

My Take

The founding document of Bayesian nonparametrics. Its genius is picking the one prior whose infinite-dimensional support is compatible with a finite-dimensional Dirichlet posterior, so conjugacy survives the jump to distribution-valued parameters. The two features that later dominated applied work — discreteness (which makes the DP a clustering prior) and the base-measure/concentration decomposition $\alpha=MG_0$ — are already here, and everything downstream in the wiki builds on this: the Pólya-urn/CRP predictive rule, Sethuraman's stick-breaking construction, Antoniak's mixtures of DPs, and the samplable DP-mixture models of Escobar-West and Neal. The one thing 1973 lacks is a constructive, simulation-friendly representation — supplied two decades later by stick-breaking.