Arlot-Celisse (2010) A Survey of Cross-Validation Procedures for Model Selection

Summary

A comprehensive survey of cross-validation (CV) for estimating an estimator's risk and for model selection. It organizes the many CV variants (hold-out, leave-one-out, leave-$p$-out, $V$-fold, Monte-Carlo) and relates their empirical behaviour to model-selection theory, carefully separating rigorous results from folklore. It analyses the bias and variance of CV risk estimates, distinguishes the estimation and identification goals, treats special frameworks (time series, many models, outliers, density estimation), and closes with practical guidelines for which CV to use when.

Key Claims

• Unified definition. CV generalizes the hold-out by averaging validation error over multiple train/validation splits; classical schemes differ by how splits are formed (exhaustive vs partial data splitting).
• Bias via the $\kappa=n/n_t$ factor. CV estimates the risk of a model trained on $n_t<n$ points, so it typically overestimates the estimation error; "unbiased risk estimation" corresponds to $\kappa\approx 1$ ($n_t\approx n$), biased estimation to $\kappa>1$.
• Variance. The variance of the CV risk estimate falls as the number of splits $B$ increases (fixed $n_t$) and grows as $n_t\to n$; for $V$-fold/LPO, where $B$ is tied to $n_t$, the variance is strongly framework-dependent and rigorously known only in special cases.
• Estimation vs identification. For estimation (efficiency/oracle) with high SNR, minimize bias ($n_t\approx n$); with low SNR, keep a small upward bias; for identification (consistency), use large bias ($n_t\ll n$). This mirrors the AIC-efficient vs BIC-consistent dichotomy.
• Framework specifics. Dependent data (time series) break the independence CV relies on and need adapted schemes; the survey also covers large model collections, robustness to outliers, and density estimation, plus closed-form CV formulas for linear estimators.
• Guidelines. Choice of CV procedure should be driven by the goal (estimation vs identification), the SNR, and the computational budget — there is no universally best CV.

Concepts Introduced or Extended

• Cross-Validation
• Model Comparison

Entities Mentioned

• Sylvain Arlot
• Alain Celisse

Quotes

"Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its (apparent) universality."

"Drawing a general conclusion on CV is nearly an impossible task because of the variety of frameworks."

My Take

The standard reference that turns "just use cross-validation" into a principled decision. Its most useful contributions are conceptual: the $\kappa=n/n_t$ bias lens, and the insistence that the right CV depends on whether you want estimation (efficiency) or identification (consistency) — the same tension that separates AIC from BIC and that reappears in the Bayesian WAIC/LOO-CV literature. For a time-series wiki the key warning is §8.1: the clean theory assumes independent data, and naive $k$-fold CV silently violates that under temporal dependence, needing block or other adapted schemes.