Afshartous-de Leeuw (2005) Prediction in Multilevel Models

Summary

Afshartous and de Leeuw (2005) fill a gap in the hierarchical linear model (HLM) literature by treating prediction of new individual observations $y_{*j}$ as a problem distinct from estimation of random effects. They define three prediction rules — Ordinary Least Squares (OLS), prior, and multilevel (shrinkage) — compare them analytically and through a large simulation (30,000 datasets), and show that the multilevel predictor dominates in 24 of 25 design conditions. A key asymmetry emerges: increasing the within-group sample size $n_j$ benefits prediction more than increasing the number of groups $J$, while the opposite holds for parameter estimation.

Key Claims

• Three prediction rules. For a new unit $*$ in group $j$, with Level 1 model $y_{ij} = X_{ij}\beta_j + \varepsilon_{ij}$ and Level 2 model $\beta_j = W_j\gamma + u_j$:
  • OLS predictor: $\hat{y}_{*j} = X_{*j}\hat{\beta}_j$ — uses only within-group data; ignores group-level information.
  • Prior predictor: $\hat{y}_{*j} = X_{*j}W_j\hat{\gamma}$ — uses only the Level 2 regression; ignores within-group fit.
  • Multilevel predictor: $\hat{y}_{*j} = X_{*j}\hat{\beta}_j^*$ where $\hat{\beta}_j^* = \Theta_j\hat{\beta}_j + (I-\Theta_j)W_j\hat{\gamma}$; the shrinkage matrix $\Theta_j = (\Sigma_\beta^{-1} + n_j\sigma^{-2}X_j'X_j)^{-1}n_j\sigma^{-2}X_j'X_j$ weights the within-group estimate toward the Level 2 prediction.
• Proposition 1.1 (Bayes optimality). The multilevel predictor equals the Bayes rule under squared error loss: $\hat{y}_{*j}^{\text{ML}} = E(y_{*j}\mid y)$.
• Proposition 1.2 (large $n_j$ limit). As $n_j \to \infty$, $\hat{\beta}_j^* \to \hat{\beta}_j$: sufficient within-group data makes shrinkage irrelevant and the multilevel predictor collapses to OLS.
• Proposition 1.3 (low Intraclass Correlation (ICC) limit). As $\rho \to 0$, $\hat{\beta}_j^* \to W_j\hat{\gamma}$: when there is no between-group variation, the multilevel predictor collapses to the prior predictor.
• Bias–shrinkage formula. If the Level 2 variance $\tau^2$ is underestimated by a factor $\upsilon < 1$, the excess shrinkage beyond the optimal is $(1-\upsilon)\tau^2/(\sigma^2 + \upsilon\tau^2)$. A 10% underestimation of $\tau^2$ (when $\tau^2=1$, $\sigma^2=1$) produces 47% additional shrinkage.
• Simulation design. 5 levels of $J$ (10, 25, 50, 100, 300) $\times$ 5 levels of $n_j$ (5, 10, 25, 50, 100) $\times$ 4 levels of ICC (0.2, 0.4, 0.6, 0.8) $\times$ 3 replication conditions = 300 design cells; 100 replications each = 30,000 datasets. Criterion: Predicted Mean Squared Error (PMSE) ratio of each predictor to the multilevel predictor.
• Multilevel predictor dominates. Multilevel wins in 24 of 25 $J \times n_j$ combinations (across ICC values). The single exception is the ($J=300$, $n_j=5$) cell, where the prior predictor wins marginally under high ICC.
• Prior predictor consistently worst. Prior prediction produces PMSE more than 1 unit above multilevel in most cells and worsens as $J$ increases — more groups does not help the pure Level 2 predictor once within-group data are available.
• Prediction vs. estimation asymmetry. Increasing $n_j$ (within-group size) reduces PMSE more effectively than increasing $J$ (number of groups); the reverse holds for estimating fixed effects $\gamma$. This divergence in optimal design strategy is the paper's central practical message.
• Misspecification robustness. Omitting a Level 2 variable from the slope equation is costlier than from the intercept-only model; the multilevel predictor remains robust under modest misspecification, especially for larger $n_j$.

Concepts Introduced or Extended

• Mixed-Effects Regression — prediction vs. estimation distinction; three prediction rules (OLS, prior, multilevel); PMSE formulas; optimal study design for prediction differs from optimal design for estimation
• Bayesian Hierarchical Model — Proposition 1.1: multilevel predictor = Bayes rule under squared error loss; shrinkage toward Level 2 prior as partial pooling
• Random Coefficient Model — shrinkage matrix $\Theta_j$ for random slopes; Proposition 1.2 large-sample collapse; bias–shrinkage formula for underestimated Level 2 variance

Entities Mentioned

• David Afshartous
• Jan de Leeuw

Quotes

"We define the multilevel predictor as the Bayes rule under squared error loss and show that it equals the best linear predictor."

"We find that the multilevel predictor outperforms the OLS and prior predictors in nearly all of the conditions investigated."

"Increasing the within-group sample size $n$ was more beneficial for prediction, while increasing the number of groups $J$ was more beneficial for estimation."

My Take

The paper's main contribution is clarifying that prediction of new observations in hierarchical models is a different problem from estimating random effects — a distinction routinely glossed over in applied multilevel work. The Bayes-optimality proof (Proposition 1.1) is clean and important: it gives a theoretical basis for preferring the multilevel predictor that goes beyond simulation evidence. The study-design asymmetry ($n_j$ for prediction, $J$ for estimation) is a practically useful rule of thumb. The main limitation is that all analytical results assume variances are known; estimation uncertainty in $\hat\tau^2$ is handled via the bias–shrinkage formula but is not propagated through a full Bayesian treatment. In practice, the bias formula shows the exposure is real: modest underestimation of Level 2 variance (common in restricted maximum likelihood in small $J$ settings) can substantially over-shrink.