Lasso

Definition

The lasso ("least absolute shrinkage and selection operator"; Tibshirani 1996) is a penalized-regression estimator that adds an $\ell_1$ penalty on the coefficients to ordinary least squares: $\hat\beta = \arg\min_\beta \sum_{i}\Big(y_i - \textstyle\sum_j x_{ij}\beta_j\Big)^2 + \lambda\sum_j|\beta_j|,$ equivalently minimizing the residual sum of squares subject to $\sum_j|\beta_j|\le t$ . Because the absolute-value penalty has non-differentiable corners on the coordinate axes, the lasso sets some coefficients exactly to zero — performing continuous shrinkage and variable selection simultaneously, a combination that neither ridge regression (no zeros) nor subset selection (unstable) achieves alone.

Key Ideas

Shrinkage + selection in one convex problem. The lasso keeps ridge's stability (continuous shrinkage of a convex objective) and subset selection's interpretability (a sparse, zeroed-out coefficient vector).
Sparsity comes from the $\ell_1$ geometry. The constraint region $\{\|\beta\|_1\le t\}$ is a cross-polytope (a diamond in 2-D) with vertices on the axes; the elliptical RSS contours typically first touch it at a corner, where some coordinates are zero. The $\ell_2$ /ridge ball is smooth and never zeros anything.
One tuning parameter. $\lambda$ (or the bound $t$ ) trades bias for variance and controls sparsity; it is chosen by cross-validation or an information criterion. As $\lambda\to 0$ the lasso approaches OLS; as $\lambda\to\infty$ all coefficients go to zero.
A penalty ladder. Subset selection ( $\ell_0$ , "how many nonzero") is combinatorial and unstable; ridge ( $\ell_2$ ) is stable but dense; the lasso ( $\ell_1$ ) is the convex relaxation that recovers sparsity while remaining tractable.

How It Works

Orthonormal design = soft-thresholding. When $X'X=I$ , the lasso solution is the soft-threshold of the OLS estimate, $\hat\beta_j=\mathrm{sign}(\hat\beta_j^{\mathrm{OLS}})\,(|\hat\beta_j^{\mathrm{OLS}}|-\lambda/2)_+$ — the same operator as Donoho–Johnstone wavelet denoising. (Ridge is proportional shrinkage; subset selection is hard thresholding.)
Computation. Tibshirani solved the constrained problem by quadratic programming over sign patterns; the whole regularization path is piecewise linear (later exploited by LARS), and cyclical coordinate descent (Friedman-Hastie-Tibshirani 2010, the glmnet algorithm) makes large- $p$ problems fast — a soft-threshold update per coordinate, warm-started down a $\lambda$ grid, extended to logistic/multinomial GLMs and the elastic net.
Bayesian interpretation. The $\ell_1$ penalty is $-\log$ of a Laplace (double-exponential) prior on the coefficients, so the lasso estimate is the posterior mode under that prior. The fully Bayesian "Bayesian lasso" represents the Laplace prior as a scale mixture of normals (normal $\times$ exponential mixing variance), giving a conjugate Gibbs sampler.

Elastic net (Zou-Hastie 2005)

The elastic net adds a ridge ( $\ell_2$ ) term to the lasso penalty: $\hat\beta=\arg\min_\beta \|y-X\beta\|^2 + \lambda_1\|\beta\|_1 + \lambda_2\|\beta\|_2^2$ (equivalently a mixing parameter $\alpha$ between pure lasso and pure ridge). It was introduced to cure two lasso deficiencies:

The $p\gg n$ saturation. The lasso can select at most $n$ variables before it saturates; the $\ell_2$ term removes this ceiling, so the elastic net can select more than $n$ predictors — essential in genomics-style $p\gg n$ problems.
The correlated-predictors problem → grouping effect. The lasso arbitrarily keeps one of a group of highly correlated predictors and drops the rest; the strictly-convex $\ell_2$ term induces a grouping effect whereby strongly correlated predictors are selected (or dropped) together with similar coefficients.
Naive vs. corrected. The "naive" elastic net applies double shrinkage (lasso then ridge), degrading prediction; Zou-Hastie rescale by $(1+\lambda_2)$ to undo the extra bias, giving the corrected elastic net.
Path algorithm. LARS-EN computes the entire elastic-net regularization path efficiently, just as LARS does for the lasso. Empirically the elastic net often out-predicts the lasso at similar sparsity.

Why It Matters

The default sparse regression. In high-dimensional problems ( $p$ large, possibly $p>n$ ) the lasso is the standard first tool for prediction and interpretation, across statistics, econometrics, and machine learning.
Anchor of a large family. It spawned the elastic net (mixes $\ell_1$ and $\ell_2$ ), the adaptive lasso (weighted penalties for selection consistency), the group lasso, the fused lasso, and non-convex penalties (SCAD, MCP) — all variations on penalized selection.
Bridge to Bayesian selection. As the mode of a Laplace prior it connects the frequentist and Bayesian (spike-and-slab, horseshoe) approaches to sparsity, and appears inside Bayesian VAR shrinkage.

Open Questions

Selection consistency requires the irrepresentable condition on the design; without it the lasso can select the wrong variables even as $n\to\infty$ .
Bias on large coefficients. The $\ell_1$ penalty over-shrinks genuinely large effects, motivating the adaptive lasso and non-convex penalties.
Correlated predictors. The lasso arbitrarily picks one of a group of correlated variables; the elastic net was introduced partly to address this.
Post-selection inference. Valid standard errors / confidence intervals after lasso selection remain delicate.