Model Comparison

Definition

Model comparison is the formal evaluation of two or more statistical models against each other, using data to determine which provides a better characterisation of the data-generating process. The appropriate criterion depends on the goal: in-sample fit (AIC, BIC, likelihood ratio), predictive accuracy (Diebold-Mariano test, log predictive score), or Bayesian evidence (Bayes factor / Marginal Data Density). In-sample fit criteria penalise for parameter count; predictive criteria evaluate out-of-sample performance; Bayesian criteria automatically trade off fit against parsimony through the prior.

Key Ideas

• Bayesian model comparison via the Marginal Data Density (MDD): $$\text{BF}_{12} = \frac{p(y \mid \mathcal{M}_1)}{p(y \mid \mathcal{M}_2)}$$ Bayes factors do not require the models to be nested and integrate out all parameters — automatically penalising complexity. See Bayes Factor and Marginal Data Density.
• Likelihood ratio test (nested models): $\text{LR} = 2[\ell(\hat\theta_1) - \ell(\hat\theta_0)] \xrightarrow{d} \chi^2_q$ under the null that $\mathcal{M}_0$ is correct ($q$ = number of restrictions).
• Information criteria: $\text{AIC} = -2\ell(\hat\theta) + 2k$; $\text{BIC} = -2\ell(\hat\theta) + k\log T$ where $k$ = number of parameters and $T$ = sample size. BIC is consistent (selects the true model asymptotically); AIC minimises predictive Mean Squared Error (MSE). See Lag-Order Selection Criteria.
• Predictive accuracy: the Diebold-Mariano (1995) test compares out-of-sample Mean Squared Forecast Error (MSFE) or log predictive scores across models without any nesting requirement. The log predictive score (log of the predictive density at the realized value) is strictly proper — it cannot be gamed — and is the Bayesian natural criterion.
• Posterior predictive checks: informal model comparison via Posterior Predictive Check — does the model's predictive distribution cover the realized data?
• Non-nested models: the Cox test and Vuong (1989) test compare likelihoods of non-nested models under regularity conditions. In the Bayesian framework, Bayes factors handle non-nested comparison naturally.

How It Works

Nested model comparison (frequentist):

1. Estimate restricted model $\mathcal{M}_0$ and unrestricted model $\mathcal{M}_1$.
2. Compute $\text{LR} = 2[\ell(\hat\theta_1) - \ell(\hat\theta_0)]$; reject $\mathcal{M}_0$ if $\text{LR} > \chi^2_{q,\alpha}$.

Information criteria (frequentist):

1. Estimate all candidate models by maximum likelihood estimation (MLE).
2. Compute AIC or BIC for each; select the model with minimum criterion.
3. AIC is preferred when prediction is the goal; BIC when consistent model identification matters.

Bayesian model comparison:

1. Place proper priors on parameters within each model.
2. Compute $\log p(y \mid \mathcal{M}_k)$ for each candidate (Chib 1995 identity, bridge sampling, or harmonic mean).
3. Rank models by posterior probability $P(\mathcal{M}_k \mid y) \propto p(y \mid \mathcal{M}_k) P(\mathcal{M}_k)$.

Out-of-sample predictive comparison:

1. Use a rolling or expanding window to produce $h$-step-ahead forecasts from each model.
2. Compare point-forecast accuracy via MSFE/Diebold-Mariano; density accuracy via log score or Continuous Ranked Probability Score (CRPS).

Why It Matters

Model comparison is the central task of empirical econometrics: without a criterion for choosing among specifications, estimation is uninformative. The choice of criterion determines the conclusion — BIC typically selects sparser models than AIC, and Bayesian Bayes factors can favour very different models than frequentist LR tests.

Open Questions

• Comparing models with very different parameter counts (e.g., Dynamic Stochastic General Equilibrium (DSGE) vs. Bayesian VAR (BVAR)) requires careful prior calibration; the Bayes factor is prior-sensitive for nested hypotheses at the boundary.
• Model averaging (Bayesian Model Averaging (BMA), equal-weight combination) often outperforms model selection in terms of predictive accuracy.
• Multiple-testing corrections when comparing many models simultaneously inflates the Type I error of pairwise tests.

Related

• Bayes Factor
• Marginal Data Density
• Lag-Order Selection Criteria
• Forecast Uncertainty Decomposition
• Forecasting
• Misspecification
• Posterior Predictive Check
• Volatility Forecast Evaluation