Betancourt-Girolami (2015) Hamiltonian Monte Carlo for Hierarchical Models

Summary

Betancourt and Girolami explain why hierarchical models are notoriously hard to sample and how Hamiltonian Monte Carlo (HMC), combined with the right parameterization, overcomes the difficulty. The exchangeable-group structure that makes hierarchical models powerful also induces a characteristic pathological geometry — the funnel — whose curvature varies so strongly that most common samplers stall. The paper reviews these pathologies, shows why naive Gibbs and Random-Walk Metropolis (and even global-covariance corrections) fail, contrasts the centered and non-centered parameterizations, and demonstrates that HMC (with NUTS, a suitable metric, and divergence monitoring) is the practical route to efficient inference. Published as a chapter in Current Trends in Bayesian Methodology with Applications (Chapman & Hall/CRC, 2015; arXiv:1312.0906).

Key Claims

• The funnel pathology. In a one-way normal model with weak data, integrating the local parameters against a global scale produces a funnel: as the log-scale $v$ (with $\tau=e^{v/2}$) decreases, the locals concentrate into a narrow region of high density but low volume (the neck); as $v$ increases, a wide region of low density but high volume (the mouth) opens. The two regions carry comparable probability mass, and the local curvature varies by orders of magnitude — any valid sampler must traverse both.
• Why naive samplers fail. Random-Walk Metropolis and the Gibbs sampler explore via steps tuned to conditional variances. Under the strong correlations of a hierarchy the conditional variances are far smaller than the marginal variances, so the chains devolve into slow random walks. Crucially, hierarchical correlations are local (position-dependent), so correcting a single global covariance / mass matrix does not remove the pathology.
• Parameterization is the key lever. The centered parameterization $\theta_i\sim N(\mu,\tau)$ suffers the funnel when the data are weak. The non-centered parameterization $\theta_i=\mu+\tau\,\eta_i$ with $\eta_i\sim N(0,1)$ makes the local variables a priori independent of the globals; when the data are weak this yields a much simpler (near-isotropic) posterior geometry. When the data are strong the centered form is better — no single parameterization dominates (Papaspiliopoulos–Roberts–Sköld 2007).
• HMC's advantage — and its own funnel struggle. HMC uses the gradient of the log-density and simulated Hamiltonian dynamics to make coherent, long-distance proposals, scaling far better than random-walk methods in high dimension. But a leapfrog integrator with a fixed step size cannot match the funnel's varying curvature: too large in the neck yields divergent trajectories, too small elsewhere is wasteful.
• Divergences as a diagnostic. Divergent numerical trajectories signal that the sampler cannot resolve the local geometry and is exploring with bias; monitoring divergences (and their dependence on step size) is essential for trustworthy inference in hierarchical models.
• Remedies. Riemannian-manifold HMC uses a position-dependent metric to adapt to local curvature; in practice, the non-centered parameterization plus NUTS plus divergence monitoring (as implemented in Stan) makes general hierarchical models tractable.

Concepts Introduced or Extended

• Non-Centered Parameterization
• Hamiltonian Monte Carlo
• Bayesian Hierarchical Model

Entities Mentioned

• Michael Betancourt
• Mark Girolami

Quotes

"When the data are sparse the density of these models looks like a 'funnel', with a region of high density but low volume below a region of low density and high volume."

"The correlations in hierarchical models, however, are not global but rather local and efficient implementations require more sophistication."

My Take

This chapter is the reference that turned "hierarchical models are hard to sample" from folklore into a precise geometric story, and it is the intellectual basis for how Stan users are taught to work today: reparameterize non-centered when the data are weak, run NUTS, and treat divergences as a red flag rather than noise. The deepest point is that the pathology is local curvature, which is why the twentieth-century fixes — rotate to decorrelate, tune a global mass matrix — don't rescue Gibbs or Random-Walk Metropolis, and why even HMC needs either a Riemannian metric or a parameterization that flattens the geometry. It pairs naturally with the mechanics in Hamiltonian Monte Carlo and with the hierarchical-modeling literature, and its centered/non-centered dichotomy is the same reparameterization idea that appears in the data-augmentation and parameter-expansion work (Gelman–Van Dyk; Liu–Wu).