Geweke-Amisano (2010) Comparing and Evaluating Bayesian Predictive Distributions of Asset Returns

Summary

Geweke and Amisano compare and evaluate Bayesian predictive distributions of asset returns, illustrated with five models applied to daily S&P 500 returns from 1976 through 2005. Bayesian inference gives exact out-of-sample predictive distributions that fully and coherently incorporate parameter uncertainty. The comparison exercise uses predictive likelihoods and is inherently Bayesian; the evaluation exercise uses the probability integral transform and is inherently frequentist. The two approaches are shown to be complementary — each identifying strengths and weaknesses in models not evident from the other.

Key Claims

• Bayesian predictive distributions. Integrating over the posterior yields exact out-of-sample predictive densities that coherently account for parameter uncertainty (no plug-in).
• Comparison via predictive likelihood. Models are ranked by their predictive likelihoods (log predictive scores); cumulative differences are log predictive Bayes factors — an inherently Bayesian, strictly proper comparison.
• Evaluation via the PIT. The probability integral transform $u_t=F_t(y_t)$ is Uniform$(0,1)$ i.i.d. under a correct predictive distribution; departures diagnose miscalibration and missed dynamics — an inherently frequentist absolute-adequacy check.
• Comparison $\ne$ evaluation. Relative ranking (which model predicts best) and absolute adequacy (is a model correctly calibrated) are different questions; a model can win the comparison yet fail the evaluation.
• Complementarity. Across five models of daily S&P 500 returns (Gaussian, GARCH, $t$-GARCH, stochastic volatility, hierarchical Markov normal mixture, 1976–2005), the Bayesian and frequentist approaches jointly reveal more than either alone.

Concepts Introduced or Extended

• Probability Integral Transform
• Model Comparison

Entities Mentioned

• John Geweke
• Gianni Amisano

Quotes

"The comparison exercise uses predictive likelihoods and is inherently Bayesian. The evaluation exercise uses the probability integral transform and is inherently frequentist. The illustration shows that the two approaches can be complementary, each identifying strengths and weaknesses in models that are not evident using the other."

My Take

A clarifying methodological point dressed as an empirical illustration: comparison and evaluation are not the same, and the Bayesian log-predictive-score answers the first while the frequentist PIT answers the second. The insistence that the winning model still be checked for absolute adequacy is the right discipline — it is exactly the gap the model-comparison machinery leaves open, since a Bayes factor only ever ranks. The Bayesian predictive densities, integrating over parameter uncertainty rather than plugging in estimates, are also the honest object to evaluate. It slots cleanly beside proper scoring rules (the comparison metric) and calibration (the reliability analogue of the PIT), and the five-model S&P 500 horse race is a compact template for density-forecast practice.