Model Selection — Cross-Validation, AIC/BIC and Bayesian LOO

Four Routes to “Which Model?”

The first question any modelling workflow faces is how complex the model should be, and raw fit cannot answer it: a more flexible model always fits the training data better, so in-sample error keeps falling as complexity grows even while the model gets worse. Every honest criterion estimates out-of-sample performance instead. There are four standard routes — cross-validation (refit on held-out folds; model-agnostic but costly), AIC and BIC (the Akaike and Bayesian information criteria — one fit each, a likelihood penalised for parameter count, differing in how much they charge per parameter), and PSIS-LOO (leave-one-out predictive accuracy estimated from a Bayesian posterior by Pareto-smoothed importance sampling).

Grading them requires knowing the answer, so the data is generated from a known cubic, $y = \tfrac12 x^3 - 2x + \varepsilon$ with $\varepsilon \sim \mathcal{N}(0,3^2)$ and $n=150$, and polynomials of degree 1–10 are fitted to it. In-sample $R^2$ rises monotonically to 0.43 at degree 10 — the trap. All four criteria select degree 3. That is where most treatments stop.

What the folds are actually measuring

One detail decides what the cross-validation is actually measuring. x is generated sorted, and scikit-learn's KFold does not shuffle by default — so cv=5 holds out five contiguous intervals of $x$, and the two outer folds are pure extrapolation beyond the training range. For polynomials that is brutal: the contiguous-fold CV error at degree 10 reaches 8,488,046 against 15 with shuffled folds. Both select degree 3 here, but only the shuffled version estimates the quantity the section claims to estimate. (Deliberately unshuffled folds are correct for dependent data — which is what purged, embargoed splits are for.)

Does the Agreement Replicate?

"All four criteria recover the truth" is a claim about one dataset. Drawing fresh samples from the same process and counting how often each lands on degree 3 separates them in a way a single draw cannot. The three cheap criteria are run on 200 draws; PSIS-LOO is reported separately on 10, because each of those draws costs a full posterior across eight polynomial degrees:

The two blocks are not equally precise and should not be read as one ranking. At 200 draws a proportion near 0.7 carries a standard error of about ±0.03; at 10 draws it is about ±0.14, so PSIS-LOO’s 0.70 is consistent with anything from roughly 0.35 to 0.93. It is reported because the direction matters — LOO tracks AIC and cross-validation rather than BIC, which is what theory predicts, since all three estimate predictive accuracy — not because 0.70 can be compared with 0.68 and 0.72 at two decimals.

That split is not three criteria failing and one working. CV, AIC and LOO all estimate predictive accuracy — and a degree-5 polynomial predicts a cubic almost exactly as well as a cubic does, so a predictive criterion has little reason to prefer the smaller model and will often take the larger. BIC asks a different question, which model generated the data, and its $k\ln n$ penalty exists to break precisely that tie. Note that no criterion ever picks too simple: the failure mode is entirely over-selection.

The sharpest version of the distinction is what more data does. BIC sits at 0.97–1.00 at every sample size tried; AIC and CV stay near 0.7 at every size tried, ending at 0.69 and 0.64 by $n$ = 2,500. More data does not rescue them, because AIC's penalty per parameter does not grow with $n$ — which is what inconsistency means. BIC is consistent, AIC is efficient, and neither property substitutes for the other. The cost of AIC's mistake is small in the currency AIC cares about and total in the currency BIC cares about.

Reading the Bayesian Comparison

The Bayesian panel needs one more correction to be read properly. The natural plot puts an error bar of $\text{SE}(\text{ELPD})$ on each model — about ±9.7 here — which makes every comparison look hopeless. (The ELPD is the expected log pointwise predictive density: the average log probability the model assigns to an unseen observation, so higher is better.) That error bar makes it look hopeless since the entire range from degree 3 to degree 8 spans two ELPD units. But that is the wrong uncertainty: two models scored on the same points make highly correlated errors, so the standard error of the difference is far smaller. az.compare reports it as dse, and it lands between 0.68 and 1.20.

Read with the paired error, LOO says something more precise and more useful than "the peak is at 3." Degrees 1 and 2 are decisively rejected — gaps of 20 and 30 ELPD against a paired SE near 6.5. Degrees 3 through 8 are not separable: gaps of 0 to 2 against paired SEs under 1.2, all flagged by arviz as $|\text{elpd\_diff}| < 4$. LOO is saying at least 3, and beyond that this sample cannot tell you — the correct answer for a criterion that targets prediction, and the same reason it picked 3 in only 7 of the 10 repeat draws (the other three went to degrees 4, 5 and 6, never lower). It behaves like AIC and unlike BIC because it is cross-validation, computed over the posterior.

The diagnostics that make the LOO numbers usable are reported rather than assumed: maximum Pareto-$k$ of 0.58 with none above 0.7, no divergences — sampler steps that failed numerically, each one a warning that the posterior's geometry defeated the sampler — and $\hat R$ between 1.00 and 1.01 across all eight posteriors, where $\hat R$ compares the spread within each chain against the spread across chains and should sit at 1.00 once they have converged on the same distribution. An ELPD from a posterior that did not converge is not worth comparing.

Where this sits

This is the criterion the Bayesian arcs already run on: the Latent Class Analysis — Choosing the Number of Classes example faces the same problem with a discrete latent dimension, and Dirichlet-Process Mixtures — How Many Components? sidesteps it by putting a prior on the count instead of scoring a grid. The over-selection pattern here is the same one that appears in Clustering — k-means, Gaussian Mixtures & Market Regimes, where BIC prefers seven regimes and a resampling check rejects them. Fold construction matters most when observations are dependent, which is the subject of Financial Returns Predictability.

Notebook

The data is simulated from a known cubic inside the notebook, so there is no data file to download and no module — the subject here is the criteria rather than an implementation.

References

• Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control 19(6), 716–723. — AIC
• Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics 6(2), 461–464. — BIC
• Shao, J. (1997). An asymptotic theory for linear model selection. Statistica Sinica 7, 221–264. — consistency versus efficiency, the distinction measured here
• Stone, M. (1977). An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion. JRSS B 39(1), 44–47. — why CV and AIC behave alike
• Watanabe, S. (2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion. JMLR 11, 3571–3594. — WAIC
• Vehtari, A., Gelman, A. & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing 27(5), 1413–1432. — PSIS-LOO, the Pareto-k diagnostic, and the standard error of the difference