Meseguer 2008 — The Out-of-Sample Performance of Stochastic Methods in Forecasting Age-Specific Mortality Rates

Summary

Meseguer (2008) — Office of Research, Evaluation, and Statistics (ORES) Working Paper No. 111 — compares the out-of-sample performance of two stochastic age-specific mortality forecasting methods: the bias-corrected Lee-Carter (LC) model and the first-order autoregressive (AR(1)) model of Denton, Feaver, and Spencer (DFS 2005). Using Human Mortality Database (HMD) data for 16 high-income countries, 21 age groups, a 1980 jump-off year, rolling-origin forecasting design, and 20,000 simulated paths per model, the paper finds that the two methods produce nearly identical point forecasts but sharply divergent interval forecasts. LC intervals are systematically too narrow — catastrophically so at retirement ages, with 0% empirical coverage in 8 of 16 countries at 16+ year horizons for dependency ratios. The AR(1) intervals are too wide but adequately calibrated. Because Old-Age, Survivors, and Disability Insurance (OASDI) solvency is sensitive to dependency ratios rather than life expectancy at birth ($e_0$), and because $e_0$ coverage is a misleading aggregate (errors cancel across ages), the paper recommends the AR(1) model for policy purposes.

Key Claims

• Point forecasts nearly identical: Root mean squared error (RMSE) ratios LC/DFS ≈ 1 across all countries and horizons; neither model dominates in point prediction accuracy.
• Both models overestimate mortality at retirement ages: Systematic bias — actual mortality improvements have exceeded model predictions at ages 65–95+. Retirement ages account for 97–99% of total age-profile mean squared error (MSE) across all countries and horizons.
• Interval forecasts diverge sharply:
  • LC: 80% intervals achieve only 50–70% empirical probability content at long horizons; 0% coverage for $\delta_2$ (old-age / total dependency ratio) in 8 of 16 countries at 16+ year horizons.
  • DFS AR(1): intervals over-calibrated but adequately capture realized outcomes; coverage at or above nominal 80% level for age-specific rates.
• $e_0$ coverage is a misleading validation criterion: Both models achieve near-100% $e_0$ coverage even when age-level intervals fail. Mortality overestimation at young ages (low variance contribution) and underestimation at old ages partially cancel in $e_0$. This "cancellation effect" is a systematic artifact — using $e_0$ as the sole metric masks age-specific interval failure.
• AR(1) preferred for OASDI policy: Dependency ratios ($\delta_1$ = old-age/working-age; $\delta_2$ = old-age/total) are the policy-relevant criterion for Social Security (SS) solvency — they determine the ratio of benefit claimants to payroll tax contributors. LC's near-zero $\delta_2$ coverage implies it provides false confidence about solvency risk.
• Mechanism of LC interval failure: LC's variance grows as $\hat{\sigma}^2 \cdot t$ along a random-walk-with-drift path for $k_t$, but the age-specific loadings $\beta_a$ do not adequately capture the covariance structure of actual improvement rates across ages. DFS's common residual covariance matrix $\hat{\Omega} = S'S/T$ more flexibly accommodates joint age-profile uncertainty.

Concepts Introduced or Extended

• SSA Mortality Forecasting — documents LC's catastrophic interval forecast failure at retirement ages and dependency ratios; establishes AR(1) as a superior policy instrument for OASDI solvency risk quantification
• Period Mortality — both stochastic models systematically overestimate mortality at ages 65–95+, connecting to the $b_x$ tilt context: the ages where mortality improvement is now most consequential are precisely where both models fail; cancellation effect shows $e_0$ is an inadequate aggregate for validation

Entities Mentioned

• Javier Meseguer
• Ronald Lee

Quotes

"The two methods produce virtually identical point forecast accuracy, but diverge markedly in interval coverage — the Lee-Carter method is consistently too narrow, especially at old ages and long horizons."

"Life expectancy at birth coverage is near 100% for both methods even when the age-specific interval coverage is essentially zero — a consequence of errors canceling across ages in the aggregation."

"For the purposes of Social Security solvency analysis, dependency ratios are the appropriate criterion, not life expectancy. On this criterion, the AR(1) model substantially outperforms the Lee-Carter model."

My Take

The paper's most important contribution is identifying the $e_0$ cancellation artifact as a methodological trap for mortality forecasting validation. Researchers who validate interval forecasts using $e_0$ alone can achieve near-perfect coverage scores while their age-specific interval forecasts are catastrophically miscalibrated — and it is precisely the age-specific failures that matter for dependency ratio uncertainty, which drives OASDI solvency risk. The retirement-age bias finding (97–99% of MSE concentrated at ages 65–95+) is also consequential for the Lee (2003) forecasting story: LC systematically underestimates the pace of old-age improvement, which is consistent with the $b_x$ tilt mechanism (improvement is migrating toward ages that LC models less precisely). The paper predates the Soneji-King (2012) Bayesian critique by four years and addresses a different failure mode — not subjective judgment vs. statistical rigor, but parametric interval miscalibration — making the two critiques complementary: Soneji-King shows SSA uses the wrong inputs; Meseguer shows the standard stochastic method uses the wrong covariance structure even if the inputs are correct.