Carvalho-Polson-Scott (2010) The Horseshoe Estimator for Sparse Signals

Summary

This is the paper that introduced the horseshoe prior, a continuous global–local shrinkage prior for the sparse normal-means problem ($y\mid\theta\sim\mathcal N(\theta,\sigma^2 I)$ with $\theta$ believed sparse). Each mean gets its own local scale with a half-Cauchy hyperprior, giving a shrinkage profile that is spiked at zero (killing noise) and heavy-tailed (leaving genuine signals almost untouched). Carvalho, Polson, and Scott motivate the estimator, prove two theorems — one characterizing its tail robustness via a score-function representation, the other establishing a super-efficient convergence rate to the true sampling density in sparse settings — and show empirically that the horseshoe closely reproduces the answers of Bayesian model averaging under a "gold-standard" point-mass (spike-and-slab) mixture, but with a fully continuous, hyperparameter-free, easy-to-sample prior.

Key Claims

• The horseshoe hierarchy. $\theta_i\mid\lambda_i\sim\mathcal N(0,\lambda_i^2)$, $\lambda_i\mid\tau\sim C^+(0,\tau)$, $\tau\mid\sigma\sim C^+(0,\sigma)$, with Jeffreys' prior $p(\sigma^2)\propto1/\sigma^2$; the estimate is the posterior mean. Each $\theta_i$ is mixed over its own local scale $\lambda_i$, with a common global scale $\tau$.
• Why "horseshoe." Writing the shrinkage weight $\kappa_i=1/(1+\lambda_i^2)$, the posterior mean is $E(\theta_i\mid y)=\{1-E(\kappa_i\mid y)\}\,y_i$, and the half-Cauchy on $\lambda_i$ implies a $\mathrm{Be}(\tfrac12,\tfrac12)$ prior on $\kappa_i$ — a U-shaped density with mass piled at $\kappa\approx0$ (no shrinkage → signal) and $\kappa\approx1$ (total shrinkage → noise).
• No user-chosen hyperparameters. The half-Cauchy priors on $\lambda_i,\tau$ and Jeffreys prior on $\sigma$ are fully specified, so the prior needs no tuning yet remains robust and adaptive across sparsity patterns.
• Tail-robustness theorem. A representation theorem characterizes a prior's tail robustness through the score function of the marginal density: heavy-tailed local scales (Cauchy) give a redescending, bounded correction so large signals are not over-shrunk — the decisive advantage over the exponential-tailed Laplace/lasso prior.
• Super-efficiency theorem. Under sparsity, the horseshoe's Bayes estimator of the sampling density converges at a super-efficient rate, driven by the prior's unbounded density at the origin — the feature that lets it shrink noise hard.
• Matches point-mass BMA. On real and simulated data the horseshoe's estimates track those of Bayesian model averaging under a two-groups point-mass mixture very closely, giving the benefits of discrete selection without the indicator variable.

Concepts Introduced or Extended

• Horseshoe Prior
• Scale Mixture of Normals

Entities Mentioned

• Carlos M. Carvalho
• Nicholas G. Polson
• James G. Scott

Quotes

"This paper proposes a new approach to sparsity, called the horseshoe estimator, which arises from a prior based on multivariate-normal scale mixtures."

"The half-Cauchy prior on $\lambda_i$ implies a horseshoe-shaped $\mathrm{Be}(1/2,1/2)$ prior for the shrinkage coefficient $\kappa_i$. The left side of the horseshoe, $\kappa_i\approx0$, yields virtually no shrinkage… and describes signals. The right side… yields near-total shrinkage and describes noise."

My Take

The horseshoe's staying power comes from getting two things right at once that earlier shrinkage priors traded off: it shrinks noise as aggressively as a spike-and-slab (an unbounded spike at the origin) while shrinking true signals as gently as a flat prior (Cauchy tails), and it does so with a continuous, tuning-free prior that drops into a Gibbs or HMC sampler. The two theorems are the paper's spine — they turn "heavy tails and a spike at zero are good" into precise statements about the score function and the convergence rate — and they explain why the horseshoe became the default continuous competitor to the lasso and spike-and-slab. For the wiki this is the primary source the horseshoe-prior page was built around; it also connects the modern global–local literature back to the classical shrinkage tradition, now with a prior engineered specifically for sparsity rather than a common mean. The practical caveats it does not fully resolve — funnel geometry in sampling, no exact zeros, sensitivity to the global scale $\tau$ — are what the later horseshoe+/regularized-horseshoe papers address.