← All examples

Machine Learning: Regularized & Kernel Learning

Two ideas run through this section, and they turn out to be the same idea seen twice.

The first is the penalty. Least squares has no unique answer when there are more predictors than observations, and a penalty supplies the missing constraint — but its shape decides what you get: an L₂ penalty shrinks everything and zeroes nothing, an L₁ penalty produces exact zeros and therefore selects. The second is the kernel, which lets a linear method act non-linearly by working with inner products in a space it never has to construct. Support vector machines are the classic instance; kernel ridge regression is the same trick applied to the penalty above.

Where they meet is Bayesian. A penalty is a prior — ridge is the MAP estimate under a Gaussian prior on the coefficients, lasso the MAP under a Laplace prior — and a kernel is a covariance function, which makes kernel ridge the posterior mean of a Gaussian process. So this section and the Bayesian Nonparametrics and Variable Selection sections are three views of one structure: the same regularisation appears as a penalty here, as a prior there, and as a posterior over which coefficients are nonzero in the third. A recurring theme is honesty about when any of it is needed — on data where n comfortably exceeds p, the penalties barely separate, and ridge can be very slightly worse than plain least squares.

Why the kernel trick is a theorem rather than a convenience. Saying a kernel lets you work in a space you never construct invites the obvious objection: if the feature space is infinite, why is the answer finite? The representer theorem is the reason. For any penalised fit of this kind, the optimal function provably lies in the span of the training points — it is a weighted sum of the kernel evaluated at the data you actually have, and nowhere else. So a model living in an infinite-dimensional space is determined by n numbers, and everything the fit needs about the feature space arrives through the Gram matrix: the n × n table of kernel values between every pair of observations. That is also the source of the cost that recurs in every example below — the Gram matrix is n² entries and inverting it is n³, which is why the kernel examples subsample where the tree examples do not. What makes a function admissible as a kernel is Mercer’s condition: it must produce a positive semi-definite Gram matrix for any inputs, which is exactly the condition for it to be an inner product in some space and for the optimisation to be convex. The space it implies is a reproducing-kernel Hilbert space, and the penalty in these methods is a norm in that space — which is the sense in which a kernel method is still just a penalised linear fit.

The one kernel that does most of the work. Nearly every kernel below is the RBF, or radial basis function, sometimes called the squared-exponential or Gaussian kernel: similarity decays with distance, k(x, x′) = exp(−γ∥x − x′∥²). Its single hyperparameter γ sets how fast that decay happens, and it is the whole model in one number. Small γ means similarity reaches a long way, so the fitted function is smooth and the method behaves almost linearly; large γ means only near-neighbours count, and the boundary can wrap around individual points. It is the same quantity a Gaussian process calls the inverse squared length-scale, which is one more place the two vocabularies describe one object — and the reason a kernel machine and a GP posterior mean land on the same number here, 0.612 against 0.611.

Choosing how much to penalise. The penalty strength λ is not estimated from the fit — it is chosen from outside, almost always by cross-validation, and the curve it produces is flatter near its bottom than people expect. Two conventions matter. The one-standard-error rule, which is glmnet’s default, does not take the λ that minimises the curve but the largest λ whose error is still within one standard error of that minimum — deliberately over-penalising, on the grounds that the minimum is itself a noisy estimate and the simpler model is the safer bet. And effective degrees of freedom turn a value of λ into something interpretable: for ridge it is the sum of d_j² / (d_j² + λ) over the design’s singular values, running from p at λ = 0 down toward zero, so it answers “how many parameters am I really fitting?” in a way the raw λ never does.

Reading the numbers. Three scores recur. AUC, the area under the receiver-operating-characteristic curve, is the probability that a randomly chosen positive case is scored above a randomly chosen negative one — a measure of ranking only, indifferent to whether the predicted probabilities are believable, with 0.5 the coin flip. RMSE, root-mean-square error, is the typical size of a prediction error in the units of the outcome, so on the California data, where the target is measured in units of $100,000, an RMSE of 0.61 is a typical miss of about six tenths of that. And the Sharpe ratio, which the two finance examples turn on, is mean excess return divided by its own standard deviation — return per unit of risk taken, annualised by √52 for the weekly portfolio data. It is worth knowing that it rewards steadiness as much as size: halving the volatility of a strategy improves it exactly as much as doubling the return.