Wood (2011) Fast Stable REML and ML Estimation of Semiparametric GLMs

Summary

Wood develops the first method for selecting the smoothing parameters of a semiparametric generalized linear model by directly optimizing a restricted maximum likelihood (REML) or marginal likelihood (ML) criterion, rather than iterating REML/ML on working linear approximations. A Laplace approximation yields an approximate REML/ML for any GLM that is a well-defined, differentiable function of the smoothing parameters, so it can be optimized efficiently and reliably. The method requires Newton-Raphson (not Fisher scoring) for the GLM fit, for which a numerically stable scheme is proposed; it is the algorithm behind the mgcv R package.

Key Claims

• Motivation (Reiss-Ogden 2009). There is a theoretical basis for sometimes preferring REML to generalized cross-validation (GCV) for smoothing-parameter selection — GCV can under-smooth and has less stable optima.
• Problem with existing REML/ML methods. They estimate smoothing parameters by iterative REML/ML on working linear approximations to the GLM (PQL-style); such indirect schemes need not converge and fail in a non-negligible fraction of analyses.
• Direct Laplace-approximate REML. A Laplace approximation gives an approximate REML/ML for any GLM suitable for direct optimization over the smoothing parameters, inheriting the reliable convergence of direct prediction-error (GCV) methods.
• Newton-Raphson requirement. The criterion requires Newton-Raphson rather than Fisher scoring for GLM fitting; the paper gives a computationally stable approach to this.
• Model class. Smooths are represented by penalized bases $f_j=\sum_k\beta_k b_k$ with quadratic wiggliness penalties $\lambda_j\beta^\top S_j\beta$ (allowing multiple penalties per term, tensor-product and adaptive bases, and general random-effect precision matrices); given $\lambda$, $\hat\beta_\lambda$ is found by penalized IRLS.

Concepts Introduced or Extended

• Generalized Additive Model
• P-splines

Entities Mentioned

• Simon N. Wood

Quotes

"This article develops the first such method for REML or ML estimation of smoothing parameters. A Laplace approximation is used to obtain an approximate REML or ML for any GLM, which is suitable for efficient direct optimization."

My Take

The paper that made REML-based GAM fitting trustworthy: by turning the marginal likelihood into a directly optimizable, differentiable function of the smoothing parameters (via a Laplace approximation, with a stable Newton-Raphson inner fit), it removed the convergence failures that plagued earlier iterative/PQL schemes. That reliability is why mgcv became the default GAM engine. The interesting tension it crystallizes — prediction-error (GCV) criteria have the better asymptotic prediction but REML is more stable and less prone to catastrophic under-smoothing — is exactly the estimation-vs-identification trade-off in another guise. It complements the basis-penalty story in P-splines: p-splines supply the smoother, Wood (2011) supplies the smoothing-parameter estimator.