Widely Applicable Information Criterion

Definition

The Widely Applicable Information Criterion (WAIC), introduced by Sumio Watanabe, is a fully Bayesian estimator of a model's out-of-sample predictive (generalization) loss that — unlike AIC and BIC — remains valid for singular statistical models (those whose parameter-to-distribution map is not one-to-one or whose Fisher information is singular). WAIC is computed from the posterior predictive density evaluated at the observed data together with a variance-based penalty for the effective number of parameters, using only draws from the posterior. Watanabe (2010) proves it is asymptotically equivalent, as a random variable, to Bayesian leave-one-out cross-validation.

Key Ideas

• "Widely applicable" = valid for singular models. A model is regular if parameters map one-to-one to distributions and the Fisher information is positive definite; most useful models (mixtures, neural nets, reduced-rank regression, HMMs, hierarchical models with hidden variables) are singular. For singular models the classical results fail: AIC ≠ average generalization error and BIC ≠ the log Bayes marginal likelihood, even asymptotically. WAIC is constructed to estimate the Bayes generalization loss regardless.
• Estimates generalization loss, not the marginal likelihood. WAIC targets predictive accuracy for new data (like cross-validation and AIC), a different goal from the marginal likelihood / Bayes factor, which targets the probability the model assigns to the observed data.
• Uses the full posterior. WAIC is averaged over the posterior (a genuinely Bayesian quantity), unlike DIC which plugs in a posterior point estimate; and it does not require the MLE, which can diverge or generalize poorly in singular models.
• The effective dimension is $\lambda$, not the parameter count. In singular learning theory the relevant complexity is the real log canonical threshold (RLCT) $\lambda$, a birational invariant from algebraic geometry that replaces the $d/2$ of BIC; it is generally smaller than the naive parameter count.

How It Works

WAIC (in Watanabe's "log loss" form) combines the log posterior-predictive fit with a functional-variance penalty: $$\mathrm{WAIC} = -\frac{1}{n}\sum_{i=1}^{n}\log\, \mathbb{E}_{\text{post}}\!\big[p(y_i\mid\theta)\big] \;+\; \frac{1}{n}\sum_{i=1}^{n}\mathrm{Var}_{\text{post}}\!\big[\log p(y_i\mid\theta)\big],$$ where the first term is the training (log) loss and the second is the functional variance $V/n$ that estimates the effective number of parameters. Both terms are computed directly from posterior draws. Watanabe (2010) establishes:

• Theorem 1: WAIC is asymptotically equivalent to the Bayes leave-one-out cross-validation loss as a random variable — so model selection / hyperparameter tuning by WAIC or by CV are asymptotically the same.
• Theorem 2: Bayes generalization error $+$ Bayes CV error $\to 2\lambda/n$, tying the CV–generalization trade-off to the RLCT $\lambda$ (neither $\lambda$ nor $n$ random).

Why It Matters

• The default Bayesian predictive criterion. WAIC (with LOO cross-validation, often via PSIS-LOO) has largely displaced DIC for comparing Bayesian models by predictive accuracy — Watanabe shows DIC is not asymptotically equivalent to CV/WAIC in singular models. Vehtari–Gelman–Gabry (2017) make LOO practical by Pareto-smoothed importance sampling (reweighting the posterior by $1/p(y_i\mid\theta)$ with tail regularization) and provide the $\hat k$ diagnostic that flags unreliable points; they find PSIS-LOO more robust than WAIC in finite samples with influential observations.
• Honest complexity for rich models. By penalizing with the functional variance / RLCT rather than the parameter count, WAIC does not over-penalize hierarchical and latent-variable models whose effective dimension is far below their nominal dimension.
• Bridges algebraic geometry and statistics. It operationalizes singular learning theory (resolution of singularities → RLCT) into a criterion practitioners can compute from MCMC output.

Open Questions

• Computing the RLCT $\lambda$ in closed form requires real algebraic geometry and is known only for special model classes; WAIC sidesteps this but the exact effective dimension is often unknown.
• WAIC's finite-sample behavior can be unstable when the posterior-predictive variance term is dominated by a few high-leverage observations (motivating PSIS-LOO diagnostics).
• Extending the asymptotic equivalence to dependent (time-series) data requires care, since the LOO construction assumes exchangeability.

Related

• Bayesian Model Selection
• Model Comparison
• Deviance Information Criterion
• Marginal Data Density
• Bayes Factor
• Proper Scoring Rule
• Importance Sampling
• Watanabe (2010)
• Sumio Watanabe
• Vehtari-Gelman-Gabry (2017) — PSIS-LOO
• Aki Vehtari
• Cross-Validation