Gelman-Meng-Stern (1996) Posterior Predictive Assessment of Model Fitness via Realized Discrepancies

Summary

Gelman, Meng, and Stern (1996) develop a unified framework for Bayesian model checking using realized discrepancies — test quantities $D(y;\theta)$ that depend on both observed data and unknown parameters, in contrast to classical test statistics $T(y)$ that depend on data alone. The posterior predictive p-value $p_b(y) = P[D(y^{rep};\theta) \geq D(y;\theta) \mid y]$ is computed as a free byproduct of Markov chain Monte Carlo (MCMC) output: simulate $y^{rep}$ alongside draws of $\theta$, then count the proportion of pairs where $D(y^{rep};\theta) > D(y;\theta)$. The paper appears in Statistica Sinica 6 (1996): 733–807, together with discussions by Rubin, Kass–Wasserman, Draper, Lewis–Raftery, and Weerahandi–Tsui.

Key Claims

• Realized discrepancy vs. test statistic: $D(y;\theta)$ depends on both data and unknown parameters; the classical $T(y)$ is the special case where dependence on $\theta$ is dropped. Including $\theta$ allows the discrepancy to target specific parameters (e.g., mean, variance, skewness), making the check more sensitive to particular forms of misfit.
• Posterior predictive p-value: $p_b(y) = P_A[D(y^{rep};\theta) \geq D(y;\theta) \mid H, y]$. Computed from MCMC as the fraction of iterations where $D(y^{rep,l};\theta^{(l)}) \geq D(y;\theta^{(l)})$; equivalently, the proportion of points above the 45° line in a scatterplot of $(D(y;\theta^{(l)}), D(y^{rep,l};\theta^{(l)}))$.
• Three replication types (Figure 1):
  • Posterior predictive (focus of paper): $y^{rep} \mid \theta$, with $\theta$ from the posterior; measures fit to observed data conditional on the current model.
  • Prior predictive (Box 1980): new $\theta$ from the prior, new $y$ — undefined for improper priors.
  • Mixed predictive: same hyperparameters $\alpha$, new $\theta$ and $y$ — appropriate for hierarchical models, avoids the improper-prior problem by conditioning on observed hyperparameters.
• Minimum discrepancy: $D_{min}(y) = \min_\theta D(y;\theta)$ is the classical analogue (analogous to $\chi^2$ or the residual sum of squares). For a normal linear model with $n$ observations and $k$ parameters: $D_{min}(y) \sim \chi^2_n - \chi^2_k$ and $D(y;\theta) \sim \chi^2_{n-k}$. Realized discrepancy yields a more conservative check because $D_{min}(y) \leq D(y;\theta)$.
• Conservatism: Under the true model, $p_b(y)$ is centered at 0.5 but is less variable than Uniform(0,1); extreme values (near 0 or 1) still indicate definitive misfit. Rubin's discussion attributes this conservatism to positive correlation between $y$ and the replications $y^{rep}$ — analogous to the superefficiency of multiple imputation where the fraction of missing information is underestimated.
• Positron emission tomography (PET) image example: $\chi^2$ realized discrepancy $\approx 30{,}000$ with $n-k = 20{,}000$ degrees of freedom; posterior predictive check (ppc) p-value $\approx 1$. The independent-Poisson model is grossly overfit (fails to account for spatial correlation in image intensities), detected unambiguously.
• Actuarial mortality (Broffitt 1988): Data are age-graduated mortality rates; the model assumes monotone increasing and convex rates. Two discrepancy statistics: $p_{min} = 10\%$ and $p_{realized} = 6.3\%$. Both indicate borderline misfit; the realized discrepancy is slightly more extreme because it penalises both data and parameter directions.
• Latent class mixture (93 infants, Stern et al. 1995): Three-class model for behavioral temperament data. Two discrepancy statistics yield ppc p-values of 93% and 74% — marginal evidence that the three-component mixture does not fully account for dependence structure across observational occasions.
• Intellectual lineage: Guttman (1967) introduced future-observation replications for goodness-of-fit; Rubin (1984) formalised the Bayesian framework with posterior predictive distributions; Gelman-Meng-Stern (GMS, 1996) unify both, extend to hierarchical models (mixed predictive), and provide the MCMC implementation.

Concepts Introduced or Extended

• Posterior Predictive Check
• Gibbs Sampler
• Mixture of Normals
• Bayesian Hierarchical Model

Entities Mentioned

• Andrew Gelman
• Xiao-Li Meng
• Hal Stern
• Donald Rubin

Quotes

"We define a 'realized discrepancy' $D(y;\theta)$ to be a scalar summary of the data $y$ and parameters $\theta$ that measures some aspect of discrepancy between model and data."

"The posterior predictive p-value $p_B(y)$ is the probability under the posterior predictive distribution that the realized discrepancy statistic for a replicated data set is more extreme than for the observed data."

"The posterior predictive p-value is conservative, in the sense that it does not have a uniform distribution under the assumed model, but is centered at 0.5 with less variance than a uniform distribution."

My Take

The paper's central insight — that letting the test statistic depend on $\theta$ makes checking essentially free once MCMC is running — is elegant and practical. The 45° scatterplot is an underappreciated diagnostic tool: it visualizes not just the p-value but also the variability and correlation structure of the check. The conservatism is a real limitation for formal testing (the p-value cannot be taken as a classical type-I error rate), but the paper's framing as model assessment rather than testing sidesteps this partially. The three-replication-type taxonomy (posterior/prior/mixed predictive) is the cleanest framework for understanding what "model checking" means in a Bayesian context. The main practical limitation is the choice of discrepancy: the framework is powerful but underdeterminate — different $D(y;\theta)$ detect different forms of misfit, and there is no omnibus statistic. The mixed predictive check for hierarchical models is an underused contribution; it properly conditions on the observed hyperparameter level and avoids the pathologies of both prior predictive (improper prior) and posterior predictive (overly conservative).