Horseshoe Prior

Definition

The horseshoe prior (Carvalho, Polson, and Scott 2010) is a continuous global–local shrinkage prior for sparse high-dimensional regression coefficients. Each coefficient is given a normal prior with its own local scale, and the scales are given heavy-tailed half-Cauchy hyperpriors: $$\beta_j\mid\lambda_j,\tau \sim \mathcal N(0,\lambda_j^2\tau^2),\qquad \lambda_j\sim C^+(0,1),\qquad \tau\sim C^+(0,1),$$ where $\tau$ is a global scale controlling overall sparsity and the $\lambda_j$ are local scales that let individual coefficients escape shrinkage. The half-Cauchy local scales produce a shrinkage profile that is simultaneously spiked at zero (aggressively shrinking noise) and heavy-tailed (leaving genuinely large signals almost untouched) — mimicking spike-and-slab behavior with a fully continuous, easy-to-sample prior.

Key Ideas

• Global–local scale mixture. The horseshoe is a scale mixture of normals: conditional on $(\lambda_j,\tau)$ the coefficient is Gaussian, so it plugs into conjugate Gibbs / HMC samplers. The global scale $\tau$ sets the baseline shrinkage level; the local scales $\lambda_j$ adapt per-coefficient.
• The "horseshoe" shape. Reparametrizing by the shrinkage weight $\kappa_j = 1/(1+\lambda_j^2)$ (with $\kappa_j\to 1$ = full shrinkage to 0, $\kappa_j\to 0$ = no shrinkage), the half-Cauchy on $\lambda_j$ induces a $\mathrm{Beta}(\tfrac12,\tfrac12)$ distribution on $\kappa_j$ — a U-shaped density with mass piled at both 0 and 1, shaped like a horseshoe. So each coefficient is pushed toward being either kept whole or killed entirely.
• Tail-robustness. The Cauchy tails of the local scales mean the prior does not over-shrink large coefficients — the key advantage over the Laplace/lasso prior, whose exponential tails bias big signals toward zero. Carvalho-Polson-Scott (2010) prove this formally via a score-function representation theorem: a heavy-tailed prior yields a bounded, redescending correction that leaves large signals almost untouched.
• Super-efficiency near the origin. The same paper proves that under sparsity the horseshoe attains a super-efficient convergence rate to the true sampling density, driven by its prior density being unbounded at zero — the feature that makes it shrink noise hard while remaining tail-robust.
• Hyperparameter-free and matches point-mass BMA. All scales have half-Cauchy/Jeffreys hyperpriors, so there are no user-chosen tuning constants; empirically the horseshoe reproduces the answers of Bayesian model averaging under a two-groups point-mass (spike-and-slab) mixture — the sparsity "gold standard" — but continuously.
• Continuous alternative to spike-and-slab. It achieves the selection-like behavior of a discrete 0/1 mixture without an indicator variable, avoiding the difficult trans-dimensional / mixing problems of spike-and-slab samplers.

Why It Matters

• Default modern sparse-Bayes prior. In high-dimensional regression (large $p$, possibly $p>n$) the horseshoe and its relatives (horseshoe+, regularized horseshoe, Dirichlet–Laplace) are the standard continuous-shrinkage choice, competitive with or better than the lasso and spike-and-slab.
• Adaptive, tuning-light shrinkage. The half-Cauchy hyperpriors adapt the amount of shrinkage to the data with little tuning, and (unlike hard selection) propagate full posterior uncertainty.
• Used in Bayesian VAR shrinkage. Global–local horseshoe priors have been adopted for shrinkage in large Bayesian VARs and sparse factor models, as an alternative to Minnesota-type or spike-and-slab shrinkage.

Open Questions

• Computation. The half-Cauchy scales create funnel-shaped posterior geometry that can be hard for samplers; slice-sampling and the regularized horseshoe were introduced partly to mitigate this.
• No exact zeros. Like all continuous-shrinkage priors it never sets coefficients exactly to zero, so a thresholding/decoupling step is needed if a genuinely sparse model is required.
• Hyperprior sensitivity. Inference on the global scale $\tau$ (and whether to fix, estimate, or regularize it) materially affects the degree of sparsity.

Related

• Spike-and-Slab Prior
• Lasso
• Scale Mixture of Normals
• Variable Selection
• Heavy-Tailed Distributions
• Bayesian VAR
• Carvalho-Polson-Scott (2010) The Horseshoe Estimator for Sparse Signals
• Hahn-Carvalho-Scott (2012)
• Carlos M. Carvalho
• Nicholas G. Polson
• James G. Scott
• James-Stein Estimator