Tibshirani (1996) Regression Shrinkage and Selection via the Lasso

Summary

This paper introduces the lasso ("least absolute shrinkage and selection operator"), the estimator that made $\ell_1$-penalized regression the default tool for simultaneous shrinkage and variable selection. The lasso minimizes the residual sum of squares subject to a bound on the sum of absolute values of the coefficients, $\sum_j|\beta_j|\le t$. Because that $\ell_1$ constraint has corners, it tends to set some coefficients exactly to zero, so — unlike ridge regression — it yields sparse, interpretable models while retaining ridge's stability. Tibshirani shows the lasso combines the good features of subset selection (interpretability, sparsity) and ridge regression (continuous, stable shrinkage), relates it to Donoho–Johnstone soft-thresholding, gives a quadratic-programming algorithm, and sketches extensions to generalized and tree-based models. (Journal of the Royal Statistical Society, Series B 58(1): 267–288.)

Key Claims

• Two failures of OLS. Ordinary least squares has low bias but high variance (poor prediction when predictors are many/collinear) and is uninterpretable when there are many predictors. One wants a smaller subset with the strongest effects.
• Why not subset selection or ridge? Subset selection gives interpretable models but is a discrete, unstable process — small data changes flip which variables are chosen, hurting prediction. Ridge is a continuous, stable shrinker but never sets coefficients to zero, so it is not interpretable. The lasso is designed to get both: continuous shrinkage and exact zeros.
• Definition. The lasso solves $\hat\beta = \arg\min_\beta \sum_i\big(y_i - \sum_j x_{ij}\beta_j\big)^2$ subject to $\sum_j|\beta_j|\le t$, equivalently the penalized form $\min_\beta \mathrm{RSS} + \lambda\sum_j|\beta_j|$. The tuning constant $t$ (or $\lambda$) controls the amount of shrinkage; small $t$ forces many coefficients to zero.
• Sparsity from the $\ell_1$ geometry. The absolute-value ($\ell_1$) constraint region is a diamond/cross-polytope whose vertices lie on the axes; the RSS contours are most likely to first touch it at a corner, where some coefficients are exactly zero — this is why the lasso selects. The quadratic ($\ell_2$/ridge) ball has no corners and never zeros coefficients.
• Orthonormal case = soft-thresholding. For an orthonormal design the lasso estimate is the soft-threshold of the OLS estimate, $\hat\beta_j = \mathrm{sign}(\hat\beta_j^{\mathrm{OLS}})\big(|\hat\beta_j^{\mathrm{OLS}}| - \gamma\big)_+$ — the same operator as Donoho–Johnstone wavelet shrinkage; ridge is proportional shrinkage and subset selection is hard-thresholding.
• Bias–variance trade. By sacrificing a little bias, the lasso reduces variance and can improve overall prediction accuracy; simulations show it does well against both subset selection and ridge across a range of designs.
• Computation and generality. The constrained problem is solved by quadratic programming (an active-set algorithm cycling over sign patterns); the idea extends readily to generalized regression (GLMs) and to tree-based models.

Concepts Introduced or Extended

• Lasso
• Variable Selection
• Scale Mixture of Normals

Entities Mentioned

• Robert Tibshirani

Quotes

"The 'lasso' minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models."

"It produces interpretable models like subset selection and exhibits the stability of ridge regression."

My Take

The lasso is one of the most consequential ideas in modern statistics precisely because of the geometric accident it exploits: the corners of the $\ell_1$ ball turn continuous shrinkage into automatic model selection, giving you ridge's stability and subset selection's interpretability in a single convex problem. For this wiki it is the frequentist twin of the Bayesian variable-selection machinery — the $\ell_1$ penalty is exactly the log of a Laplace (double-exponential) prior, so the lasso is the posterior mode under that prior, and its Bayesian counterpart (the Bayesian lasso) is a scale-mixture-of-normals Gibbs sampler. It sits opposite spike-and-slab (a genuine 0/not-0 mixture) as the two dominant approaches to sparsity, and its soft-thresholding identity ties it to wavelet denoising. The main caveats — inconsistent selection without the irrepresentable condition, and bias on large coefficients — motivated the adaptive lasso, elastic net, and non-convex penalties that followed, but they are all descendants of this paper.