Generalized Additive Model

Definition

A generalized additive model (GAM) is a generalized linear model in which the linear predictor is replaced by a sum of smooth functions of covariates: $g(\mu_i)=\sum_j f_j(x_{ji})$, where $\mu_i=E[y_i]$, $g$ is a link, and each $f_j$ is an unknown smooth term. The smooths are represented by basis expansions with wiggliness penalties, and the amount of smoothing is controlled by penalty (smoothing) parameters chosen from the data (Wood 2011).

Key Ideas

• Basis-penalty representation. Each smooth $f_j$ is written $f_j(x)=\sum_k \beta_k b_k(x)$ in a chosen basis (e.g. P-splines, thin-plate/cubic regression splines, tensor products for interactions), with a quadratic penalty $\lambda_j\,\beta^\top S_j\beta$ measuring wiggliness; the $S_j$ are fixed positive-semidefinite matrices and $\lambda_j\ge0$ the smoothing parameters.
• Penalized fitting given $\lambda$. For fixed smoothing parameters the penalized log-likelihood has a unique maximizer $\hat\beta_\lambda$, computed by penalized iteratively reweighted least squares (PIRLS) — the penalized analogue of GLM estimation.
• Smoothing-parameter selection is the crux. Choosing $\lambda$ trades fit against smoothness. Two schools:
  • Prediction-error criteria: minimize AIC, cross-validation, or generalized cross-validation (GCV) — asymptotically best for prediction but prone to occasional severe under-smoothing and multiple/unstable minima.
  • Likelihood/random-effects criteria: treat the smooths as random effects (penalty $\leftrightarrow$ Gaussian prior, $\lambda_j\leftrightarrow$ variance component) and estimate $\lambda$ by marginal likelihood (ML) or restricted maximum likelihood (REML) — the empirical-Bayes view.
• Bayesian foundation (Wahba 1978). The random-effects view rests on a classical equivalence: (generalized) spline smoothing is Bayesian estimation under a partially improper prior — a flat (improper) prior on the parametric span of the regression functions and a proper Gaussian (roughness-penalty) prior on the deviation from that span. This frames smoothing as hedging against model error: rather than trust that the truth lies in the span of a fixed set of regression functions, the smooth adds a data-estimated deviation, and the smoothing parameter is the prior-to-error variance ratio. Kimeldorf-Wahba (1970) and Wahba (1978) are the roots of the penalty$\leftrightarrow$prior, $\lambda\leftrightarrow$variance-component correspondence that REML exploits.
• Why REML (Wood 2011). Reiss-Ogden (2009) give a theoretical basis for preferring REML to GCV; Wood provides the first method that directly optimizes a Laplace-approximate REML/ML over the smoothing parameters of a GLM, rather than iterating REML on working linear approximations (which need not converge). It requires Newton-Raphson (not Fisher scoring) for the GLM fit, with a numerically stable implementation — the algorithm behind the mgcv R package.

How It Works

Pick bases and penalties for each term, forming the model matrix and penalty matrices $S_j$. Outer loop: optimize the chosen smoothness criterion $V(\lambda)$ — GCV or the Laplace-approximate REML — over $\log\lambda$. Inner loop: for each trial $\lambda$, run PIRLS to get $\hat\beta_\lambda$. Wood's contribution is making the REML objective a properly-defined, differentiable function of $\lambda$ that can be optimized directly and stably, so smoothing-parameter estimation converges reliably where earlier indirect schemes failed.

Why It Matters

• Flexible mean structure with automatic smoothness. GAMs capture non-linear covariate effects without pre-specifying functional form, while the penalty prevents over-fitting — the workhorse of applied semiparametric regression.
• Reliable, reproducible fitting. Direct REML optimization fixed the convergence failures of earlier GAM software, making mgcv a de facto standard across ecology, epidemiology, and time-series/spatial smoothing.
• Unifies smoothing and mixed models. The random-effects view connects penalized smoothing to variance-component estimation and empirical Bayes, letting GAMs and mixed models be fitted in one framework.

Open Questions

• REML vs prediction-error selection: REML is more stable, prediction-error criteria have better asymptotic prediction — when each is preferable remains context-dependent.
• Uncertainty quantification for the smooths that accounts for smoothing-parameter estimation (not just conditional-on-$\lambda$ intervals).
• Scaling to very large data and high-dimensional tensor-product interactions.

Related

• P-splines
• Cross-Validation
• Laplace Approximation
• Link Function
• Wood (2011) Fast Stable REML and ML Estimation of Semiparametric GLMs
• Wahba (1978) Improper Priors, Spline Smoothing and Guarding Against Model Errors
• Simon N. Wood
• Grace Wahba
• Trevor Hastie