Definition
High-dimensional inference concerns estimation and — especially — valid uncertainty quantification when the number of parameters p is comparable to or far exceeds the sample size n (p≫n). In the sparse linear model Y=Xβ+ε with only a few nonzero coefficients, point estimation is handled by penalized methods such as the Lasso, but producing trustworthy p-values and confidence intervals for individual coefficients requires additional machinery, because selecting and testing on the same data invalidates classical standard errors (Bühlmann-Kalisch-Meier 2014).
Key Ideas
- Sparsity makes p≫n tractable. Ordinary least squares is unidentified and overfits when p>n; assuming most βj=0 restores a well-posed problem and licenses ℓ1-penalized estimation.
- Selection = inference. A model chosen by the Lasso carries no valid p-values: the data were used twice (to select and to test), so naive post-selection intervals are anticonservative. This is the central problem the field addresses.
- Sample splitting / multi-splitting. Split the sample, select variables on one part and compute p-values on the other; aggregating over many random splits (multi-sample-splitting) gives reproducible p-values that control the family-wise error rate.
- Stability selection. Repeatedly subsample and track each variable's selection frequency; retain only variables selected often. This controls the expected number of false positives under weak assumptions and is far more robust than a single regularization-path fit.
- De-sparsified (debiased) Lasso. Remove the Lasso's shrinkage bias to obtain an asymptotically normal estimator for each βj, yielding coordinate-wise p-values and confidence intervals valid even when p≫n; a scaled Lasso estimates the noise variance jointly.
- Multiple-testing control. With many hypotheses, control the family-wise error rate (FWER) or the false discovery rate (FDR) — the expected fraction of false rejections among all rejections — rather than individual type-I error.
- Underlying conditions. Guarantees rest on sparsity plus design conditions (restricted eigenvalue / compatibility) and often a minimal-signal ("beta-min") assumption; these are generally unverifiable in practice.
How It Works
- Fit a sparse estimator (Lasso, scaled Lasso) to obtain a candidate active set and a noise-level estimate.
- Choose an inference strategy: multi-sample-splitting or stability selection for robust selection with error control, or the de-sparsified Lasso for coordinate-wise p-values and confidence intervals.
- Apply a multiple-testing correction (FWER or FDR) across the many coefficients tested.
- Report selected variables with calibrated significance, acknowledging dependence on the sparsity/design assumptions.
Why It Matters
- Honest discovery in genomics and beyond. High-throughput biology routinely has thousands of covariates and dozens of samples; distinguishing real signals from artifacts requires exactly this valid, multiplicity-aware inference.
- Completes the sparse-regression toolkit. It supplies the missing inferential layer above Lasso-type point estimation, turning "which variables were selected" into "which variables are significant, and how sure are we."
- A frequentist mirror of Bayesian sparsity. It answers the same "which variables, how sure?" question as spike-and-slab posterior inclusion probabilities and horseshoe shrinkage, by a different route.
Open Questions
- Unverifiable assumptions. Compatibility/restricted-eigenvalue and beta-min conditions cannot be checked from data, so the guarantees are conditional.
- Power vs. error control. Methods that rigorously control false positives can be low-powered; the trade-off in realistic, correlated designs is not fully settled.
- Beyond linear/Gaussian. Extending valid high-dimensional inference to generalized linear models, dependent data, and graphical/causal structures is an active area.
Related