Watanabe (2010) Asymptotic Equivalence of Bayes Cross-Validation and Widely Applicable Information Criterion in Singular Learning Theory

Summary

This paper establishes the theoretical foundation for the Widely Applicable Information Criterion (WAIC) and its equivalence to Bayesian leave-one-out cross-validation for singular statistical models. A model is regular if the parameter-to-distribution map is one-to-one and the Fisher information is positive definite; it is singular otherwise. Most modern learning machines — neural networks, mixtures, reduced-rank regression, hidden Markov models, and generally any model with hidden variables or hierarchical structure — are singular, and for them the classical AIC/BIC and MLE asymptotics break down. Using singular learning theory (algebraic geometry / resolution of singularities), Watanabe proves two results: (1) the Bayes cross-validation loss is asymptotically equivalent to WAIC as a random variable, so model selection or hyperparameter tuning by either is asymptotically the same; and (2) the sum of the Bayes generalization error and the Bayes cross-validation error is asymptotically $2\lambda/n$, where $\lambda$ is the real log canonical threshold (RLCT) — a birational invariant of the model — and $n$ the sample size. (Journal of Machine Learning Research 11: 3571–3594.)

Key Claims

• Regular vs. singular models. Regular: the map from parameters to distributions is injective and the Fisher information matrix is positive definite. Singular: not regular — which includes neural nets, normal mixtures, reduced-rank regression, Boltzmann machines, HMMs, and any model with hidden variables, hierarchy, or grammatical structure. Singularity is the rule, not the exception, in machine learning.
• Classical criteria fail under singularity. For singular models, AIC is not the average generalization error, BIC is not the (log) Bayes marginal likelihood — even asymptotically — and the MLE may diverge or generalize poorly. Bayes estimation, by contrast, provably lowers generalization error when the model is singular, so the paper studies estimating the Bayes generalization error.
• Singular learning theory. Via Hironaka's resolution of singularities, the log-likelihood is put in a standard (normal-crossing) form even when singular. The asymptotic Bayes marginal likelihood is then governed by the real log canonical threshold $\lambda$ (generalizing the $d/2$ of BIC), and the average Bayes generalization error is estimable by WAIC (Watanabe 2009).
• Theorem 1 — CV ≡ WAIC. The Bayes cross-validation loss is asymptotically equivalent to WAIC as a random variable (not merely in expectation). Hence model selection and hyperparameter optimization by Bayes LOO-CV and by WAIC are asymptotically identical.
• Theorem 2 — the $2\lambda/n$ law. The sum of the Bayes generalization error and the Bayes cross-validation error is asymptotically $2\lambda/n$. Since $\lambda$ (the RLCT) is a birational invariant determined by the model's algebraic-geometric structure and neither $\lambda$ nor $n$ is random, the trade-off between in-sample CV error and out-of-sample generalization error is fixed by that structure.
• WAIC/CV ≠ DIC. The deviance information criterion (DIC) is not asymptotically equivalent to Bayes cross-validation or WAIC in singular models — a caution against using DIC as if it estimated generalization error there.

Concepts Introduced or Extended

• Widely Applicable Information Criterion
• Bayesian Model Selection
• Model Comparison

Entities Mentioned

• Sumio Watanabe

Quotes

"In regular statistical models, the leave-one-out cross-validation is asymptotically equivalent to the Akaike information criterion. However, since many learning machines are singular statistical models, the asymptotic behavior of the cross-validation remains unknown."

"The sum of the Bayes generalization error and the Bayes cross-validation error is asymptotically equal to $2\lambda/n$, where $\lambda$ is the real log canonical threshold and $n$ is the number of training samples."

My Take

This paper is the rigorous justification for why WAIC has largely displaced DIC as the default Bayesian predictive model-selection criterion: it proves WAIC estimates the same thing as Bayes cross-validation — the out-of-sample generalization loss — and does so without the regularity assumptions that AIC/BIC quietly require and that essentially every interesting Bayesian model violates. The deep and surprising content is that the whole CV-vs-generalization trade-off collapses to a single number, the real log canonical threshold $\lambda$, an invariant from algebraic geometry — so the "effective dimension" of a singular model is $\lambda$, not the naive parameter count. For applied Bayesian work the takeaways are practical: prefer WAIC (or PSIS-LOO) over DIC for predictive comparison, and expect the parameter count to overstate model complexity whenever the model has hidden variables or hierarchy. The main barrier is that computing $\lambda$ in closed form requires real algebraic geometry; in practice WAIC is used precisely because it estimates the relevant quantity without needing $\lambda$ explicitly.