Posterior Predictive Check

Definition

A posterior predictive check (PPC) is a Bayesian model diagnostic that asks: does replicated data $y^{rep}$ generated from the fitted model look like the observed data $y$ ? The check is operationalised via a realized discrepancy $D(y;\theta)$ — a scalar measure of discrepancy that depends on both data and unknown parameters $\theta$ . The posterior predictive p-value is $p_b(y) = P[D(y^{rep};\theta) \geq D(y;\theta) \mid y]$ , where both $\theta$ and $y^{rep}$ are drawn from their joint posterior predictive distribution.

Key Ideas

Realized vs. classical discrepancy: Classical test statistics $T(y)$ depend on data alone; realized discrepancies $D(y;\theta)$ additionally depend on $\theta$ , allowing the check to target specific features of the model (e.g., a discrepancy involving the residual variance tests variance fit, not mean fit).
Markov Chain Monte Carlo (MCMC) computation is free: At each MCMC iteration $l$ , draw $y^{rep,l} \sim p(y^{rep} \mid \theta^{(l)})$ alongside the parameter draw. The p-value is the fraction of iterations where $D(y^{rep,l};\theta^{(l)}) \geq D(y;\theta^{(l)})$ ; equivalently, the proportion of points above the 45° line in the $(D(y;\theta^{(l)}),\, D(y^{rep,l};\theta^{(l)}))$ scatterplot.
Three replication types: Posterior predictive (same $\theta$ from posterior, new $y^{rep}$ ; standard check), prior predictive (new $\theta$ from prior, new $y$ ; Box 1980, undefined for improper priors), and mixed predictive (same hyperparameters $\alpha$ , new $\theta$ and $y$ ; appropriate for hierarchical models, avoids improper-prior pathology).
Conservatism: Under the true model, $p_b(y) \sim \text{something centered at } 0.5$ but with less dispersion than Uniform(0,1). This means the ppc p-value cannot be used directly as a type-I error probability, but extreme values (near 0 or 1) still constitute strong evidence of misfit.
Minimum discrepancy (classical analogue): $D_{min}(y) = \min_\theta D(y;\theta)$ recovers classical goodness-of-fit statistics. For a normal linear model ( $n$ obs, $k$ params): reference distribution of $D_{min}$ is $\chi^2_n - \chi^2_k$ , while the realized $D(y;\theta) \sim \chi^2_{n-k}$ — the realized check is always more conservative ( $D_{min} \leq D$ ).

How It Works

Fit the model by MCMC, obtaining draws $\{\theta^{(l)}\}_{l=1}^L$ .
At each iteration, simulate $y^{rep,l} \sim p(y^{rep} \mid \theta^{(l)})$ .
Compute $D(y;\theta^{(l)})$ and $D(y^{rep,l};\theta^{(l)})$ .
Plot the pairs as a scatterplot with the 45° reference line.
$\hat p_b = L^{-1} \sum_l \mathbf{1}[D(y^{rep,l};\theta^{(l)}) \geq D(y;\theta^{(l)})]$ .

Values near 0 or 1 indicate the observed data are extreme relative to what the model predicts — either the chosen discrepancy statistic is systematically smaller (near 0) or larger (near 1) for observed than for replicated data. Values near 0.5 indicate adequate fit along the dimension measured by $D$ .

Application: Long-Horizon Event Studies (Brav 2000)

Brav (2000) applies the predictive framework to inference in long-horizon buy-and-hold return studies. The discrepancy is the sample mean abnormal return; the predictive distribution is constructed by simulating buy-and-hold returns from the posterior of a seemingly unrelated regression (SUR) model with shrinkage priors on firm residual standard deviations (SDs) and a common equicorrelation $\rho$ . This is formally a posterior predictive check — the observed sample mean is compared against the predictive null density — and it dominates the nonparametric bootstrap by correctly propagating both residual variance uncertainty and cross-sectional dependence. Applied to 1,521 initial public offerings (IPOs, 1975–1984), the characteristic-based model cannot be rejected while the Fama-French three-factor model is rejected.

Why It Matters

PPCs provide a practically free model diagnostic once MCMC is running, with no additional model fitting required. They can be tailored to any aspect of concern: autocorrelation structure, tail behavior, conditional means, variance homogeneity. In hierarchical models, the mixed predictive variant is particularly appropriate because it conditions on observed group-level effects and avoids the extreme conservatism of the posterior predictive check at the hyperparameter level. The framework unifies Guttman's (1967) future-observation replications, Rubin's (1984) Bayesian predictive distributions, and classical goodness-of-fit into a single computational recipe.

Open Questions

Choice of discrepancy: No omnibus $D(y;\theta)$ exists; different statistics detect different forms of misfit and the analyst must choose in advance. This makes PPCs confirmatory rather than exploratory for many applications.
Calibration: Because ppc p-values are conservative (not uniform under the true model), formal multiple-testing corrections cannot be straightforwardly applied.
Relationship to Bayes factors: Gelman-Meng-Stern acknowledge that Bayes factors provide a more principled model-comparison tool, but PPCs are computationally cheaper and applicable with improper priors (posterior predictive variant).