Vehtari-Gelman-Gabry (2017) Practical Bayesian Model Evaluation Using Leave-One-Out Cross-Validation and WAIC

Summary

This paper gives the practical, computable recipe for estimating a Bayesian model's out-of-sample predictive accuracy from posterior draws, via leave-one-out cross-validation (LOO) and WAIC. Both estimate the pointwise expected log predictive density (elpd) using the log-likelihood evaluated at the posterior simulations — no model re-fitting needed. Exact LOO would require re-fitting $n$ times; approximate LOO can be had by importance sampling (reweighting the full posterior by $1/p(y_i\mid\theta)$), but those weights have heavy tails and can have infinite variance. The paper's central contribution is Pareto-smoothed importance sampling (PSIS), which fits a generalized Pareto distribution to the upper tail of the importance weights and replaces the extreme weights with smoothed values — stabilizing the estimate and, crucially, providing the $\hat k$ diagnostic that flags when the approximation is unreliable. PSIS-LOO is shown to be more robust than WAIC in finite samples with weak priors or influential observations. The methods are implemented in the R package loo and demonstrated with Stan. (Statistics and Computing 27(5): 1413–1432.)

Key Claims

• The target: elpd. Predictive accuracy is measured by the expected log pointwise predictive density, $\mathrm{elpd}=\sum_{i=1}^n\int p_t(\tilde y_i)\log p(\tilde y_i\mid y)\,d\tilde y_i$, where $p_t$ is the (unknown) true data-generating distribution; cross-validation and WAIC are ways to estimate it from within-sample fits.
• LOO by importance sampling. The leave-one-out predictive density can be estimated from the full posterior draws by reweighting each draw for point $i$ by the ratio $w_i^s\propto 1/p(y_i\mid\theta^s)$ (Gelfand–Dey–Chang) — avoiding $n$ re-fits — but these importance weights are heavy-tailed, so the raw estimate is noisy and can have infinite variance (Peruggia 1997).
• PSIS — the fix. Pareto-smoothed importance sampling fits a generalized Pareto distribution to the largest importance weights (the upper tail) and replaces them with the fitted order-statistic expectations, regularizing the weights. This yields a far more stable and accurate LOO estimate from the same posterior draws.
• The $\hat k$ diagnostic. The estimated Pareto shape parameter $\hat k$ for each observation diagnoses reliability: $\hat k<0.5$ is fine, $0.5\le\hat k<0.7$ is usable, and $\hat k>0.7$ signals that the importance-sampling approximation is untrustworthy for that point — in which case exact LOO or K-fold CV should be used there. This per-point warning is a key practical advantage.
• PSIS-LOO vs. WAIC. WAIC is asymptotically equal to LOO, but PSIS-LOO is more robust in finite samples with weak priors or influential observations; both come with diagnostics for when they fail.
• WAIC beats DIC. WAIC is fully Bayesian (uses the whole posterior), invariant to parametrization, and works for singular models — unlike DIC, which is based on a point estimate, can give a negative effective number of parameters, and is undefined for singular models.
• Model comparison with standard errors. As a byproduct, the pointwise computation yields approximate standard errors for the estimated elpd and for the difference in elpd between two models, enabling honest model comparison. Everything is packaged in the R loo library.

Concepts Introduced or Extended

• Widely Applicable Information Criterion
• Importance Sampling
• Model Comparison
• Bayesian Model Selection

Entities Mentioned

• Aki Vehtari
• Andrew Gelman
• Jonah Gabry

Quotes

"We introduce an efficient computation of LOO using Pareto-smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. Although WAIC is asymptotically equal to LOO, we demonstrate that PSIS-LOO is more robust in the finite case with weak priors or influential observations."

"WAIC is fully Bayesian in that it uses the entire posterior distribution, and it is asymptotically equal to Bayesian cross-validation. Unlike DIC, WAIC is invariant to parametrization and also works for singular models."

My Take

If Watanabe supplied the theory that WAIC estimates cross-validated predictive loss, this paper supplied the engineering that made cross-validation itself cheap and safe: PSIS turns the appealing-but-unstable "reweight the posterior" LOO into a routine, diagnosable computation, and the $\hat k$ statistic is the part practitioners actually rely on — a built-in alarm that tells you exactly which observations break the approximation. For this wiki it is the applied companion to the WAIC page and a nice illustration of importance sampling done carefully (tail regularization rather than blind reweighting). The loo package it introduced became the default model-comparison tool in the Stan ecosystem, and its "compare elpd differences with standard errors" workflow largely displaced information-criterion point comparisons. The honest limitation, which the authors foreground, is that when $\hat k$ is large the shortcut fails and you must pay for K-fold or exact LOO.