Shiller (1984) Smoothness Priors and Nonlinear Regression

Summary

Develops Bayesian smoothness priors for nonlinear regression — priors that express the belief that an unknown function $f(x)$ changes slope gradually (i.e., is smooth). The method allows estimation of an arbitrary unknown function with standard linear regression software: the smoothness prior is equivalent to adding dummy observations that penalize curvature, so no new algorithms are needed. Connects Bayesian smoothness priors to cubic spline interpolation (the posterior mean under continuous smoothness priors is the cubic natural spline) and to earlier distributed-lag shrinkage priors (Shiller 1973). Illustrated on Nerlove (1963) electricity generation cost data.

Key Claims

• Setup. Model: $y_i = f(x_i) + \mathbf{z}_i'\boldsymbol\gamma + \varepsilon_i$ where $f(\cdot)$ is an unknown smooth function of one regressor $x$, $\mathbf{z}_i$ contains additional linear regressors, and $\varepsilon_i \sim N(0, h^{-1})$. Estimate $\mathbf{f} = (f(\tilde{x}_1),\ldots,f(\tilde{x}_N))'$ at $N$ chosen gridpoints (which include the observed $x_i$ values and possibly additional interpolation points).
• Continuous smoothness prior. Based on Wahba (1978): the prior variance-covariance of $\mathbf{f}$ is proportional to $H^{-1} = (RQ_0R')^{-1}$, where $R$ is the $(N-2)\times N$ second-difference matrix. The prior penalizes curvature: $p(\boldsymbol\beta, h) \propto h^{N-g/2}\exp(-kh\,\boldsymbol\beta'R'H^{-1}R\boldsymbol\beta/2)$, where $k$ is the smoothness precision and $g$ is the number of additional linear regressors.
• Discrete smoothness prior. Simpler variant (Shiller 1973, Whittaker-Robinson 1967): prior directly penalizes discrete second differences $\Delta^2 f(\tilde{x}_i) = f(\tilde{x}_{i+1}) - 2f(\tilde{x}_i) + f(\tilde{x}_{i-1})$. Computationally equivalent but does not require the continuous-derivative construction; most useful when gridpoints take on integer values only.
• Dummy-observation implementation. Both prior types reduce to adding rows to the design matrix: the prior $(6)$ is equivalent to adding $N-2$ dummy observations that set the second differences of $\mathbf{f}$ to zero with weight $k$. Estimation uses standard ordinary least squares (OLS) on the augmented $[y; \mathbf{0}]$ system with augmented design matrix $[X; \sqrt{k} R'H^{-1/2}]$. Coefficient estimates and standard errors fall out directly — no new software required.
• Bayesian interpretation. The smoothing parameter $k$ has a Bayesian interpretation: it is the precision (inverse variance) of the prior on curvature. Large $k$ $\to$ smooth (near-polynomial) curve; $k \to 0$ $\to$ function passes exactly through data. As $k \to \infty$, the estimator converges to OLS projection onto a constant, $x$, and $x^2$ (a second-degree polynomial).
• Connection to cubic splines (Proposition 3). With continuous smoothness priors, if the gridpoint $\tilde{x}_i$ does not correspond to an observation, the posterior mean for $f(\tilde{x}_i)$ is the value of the cubic natural spline that minimizes the integrated squared second derivative subject to interpolating the data. The smoothing parameter $k$ directly controls the spline roughness penalty.
• Invariance property (Proposition 2). The posterior mean and standard errors for $\mathbf{f}$ are unchanged by adding or removing prediction gridpoints that do not correspond to observations — the estimate is globally coherent. This property holds for continuous but not discrete smoothness priors.
• Application. Nerlove (1963) cost data ($n$ electricity generating firms, cost per unit vs. output). Continuous smoothness prior with $k=1$ gives a declining-then-flat cost curve (costs fall for small firms, stabilize for large firms). Fifth-degree polynomial regression gives a similar curve but shows a spurious upturn at the upper end — a known artifact of polynomial regression's inability to represent asymptotes.

Concepts Introduced or Extended

• (no existing wiki concept pages for nonparametric smoothing or shrinkage priors)

Entities Mentioned

• (none with existing wiki pages)

Quotes

"In most applications of the linear regression model, there is apparently no theory that the relation should be linear. Linearity is assumed for simplicity."

"Those who use the multiple-regression model in applied work often find themselves seeking simple expedients to allow for nonlinearity in an independent variable. This article offers another such simple expedient based on recent Bayesian literature on spline smoothing."

My Take

An elegant bridging paper connecting three literatures: Bayesian shrinkage priors (Shiller 1973), spline smoothing (Wahba 1978), and standard regression practice. The main practical contribution is the dummy-observation implementation — it means any econometrician with an OLS package can do Bayesian nonparametric curve fitting without special software. The connection to cubic splines (Proposition 3) gives a firm frequentist/nonparametric interpretation of the Bayesian estimator and helps calibrate $k$. Historically important as a precursor to the ridge regression and Tikhonov regularization literature applied to function estimation. In time-series settings, the discrete smoothness prior (Shiller 1973) is used for smooth distributed lag estimation — the same framework as polynomial distributed lags but without requiring the polynomial form. The paper does not address the data-driven selection of $k$ (cross-validation or empirical Bayes), which would become a major focus of subsequent literature.