P-splines

Definition

A P-spline (penalized B-spline) is a nonparametric smoother that fits a curve as a linear combination of a rich B-spline basis — many equidistant knots — while a difference penalty on adjacent basis coefficients controls smoothness (Eilers-Marx 1996). By penalizing the coefficients rather than choosing knot positions, it decouples the hard knot-selection problem from the smoothness problem: use plenty of knots and let a single penalty parameter $\lambda$ regulate the fit.

Key Ideas

• Basis + penalty. Model $\mu = B a$ where $B$ is an $m\times n$ B-spline design matrix over equally spaced knots and $a$ the coefficient vector. Smoothness is imposed by a $k$-th order difference penalty $\lambda\|D_k a\|^2$, where $D_k$ forms $k$-th differences of adjacent coefficients ($D_2 a_j = a_j-2a_{j-1}+a_{j-2}$).
• Penalized least squares. $\hat a = (B'B + \lambda D_k'D_k)^{-1}B'y$; the fit is linear in $y$ with smoother ("hat") matrix $H = B(B'B+\lambda D_k'D_k)^{-1}B'$.
• Connection to smoothing splines. For second-order differences the difference penalty is a discrete analogue of O'Sullivan's / Reinsch's integrated-squared-second-derivative penalty — the classical smoothing-spline penalty — but far simpler to compute and generalize to any difference order.
• Effective dimension = trace of the smoother. The effective number of parameters is $\mathrm{tr}(H)$ (following Hastie–Tibshirani), decreasing smoothly from $n$ (at $\lambda=0$) toward the order of the penalty polynomial as $\lambda\to\infty$. This scalar makes different knot counts, spline degrees and penalty orders directly comparable.
• Choosing $\lambda$. Cross-validation or AIC built on the effective dimension; both are cheap because everything is a (generalized) linear regression.
• Limit and conservation properties. As $\lambda\to\infty$ the fit approaches a polynomial of degree $k-1$; P-splines conserve moments (means, variances) of the data and have no boundary effects, unlike kernel smoothers.
• GLM / likelihood extension. Drop the penalty into a penalized likelihood and solve by iteratively reweighted least squares — giving penalized nonparametric logistic regression, Poisson smoothing, and density estimation (smoothing a fine histogram as Poisson counts).

How It Works

Over-parameterize deliberately: place, say, 10–40 equidistant knots regardless of the true smoothness. The unpenalized fit would overfit, so add $\lambda\|D_k a\|^2$, which shrinks differences between neighbouring coefficients toward zero — a locally smooth coefficient sequence produces a smooth curve. Because the penalty acts on coefficients (not an integral of the curve), the normal equations gain only the small $\lambda D_k'D_k$ term, so P-splines slot into any software that already does B-spline regression. Tune $\lambda$ by minimizing AIC/CV over the effective dimension $\mathrm{tr}(H)$.

Why It Matters

• Removes knot-placement as a modelling burden, the historical Achilles heel of regression splines — a single continuous $\lambda$ replaces a discrete combinatorial search.
• General-purpose smoother for scatterplot smoothing, nonparametric logistic/Poisson regression, and density estimation, and a natural building block for generalized additive models and, with tensor products, multidimensional and spatial smoothing.
• Cheap and modular: standard (generalized) linear-regression machinery, closed-form linear smoother, trivial effective-dimension bookkeeping.

Open Questions

• Automatic, robust selection of $\lambda$ (CV vs AIC vs mixed-model REML) and its behaviour under correlated errors.
• Choice of penalty order $k$ and number of knots — largely conventional rather than principled.
• Extending the trace-based effective dimension cleanly to the generalized (IRLS) and multidimensional/tensor-product cases.

Related

• Lasso
• Eilers-Marx (1996) Flexible Smoothing with B-splines and Penalties
• Paul H. C. Eilers
• Brian D. Marx
• Trevor Hastie
• Reversible Jump MCMC
• Generalized Additive Model
• Wahba (1978) Improper Priors, Spline Smoothing and Guarding Against Model Errors
• Grace Wahba
• Frailty Model