Support Vector Machines & Kernel Methods¶
Max-margin classification · the kernel trick · kernel ridge = Gaussian process¶
The regularized-linear notebook penalised coefficient size; this one changes the loss and then goes nonlinear. A support vector machine replaces squared error with the hinge loss and seeks the separating hyperplane with the widest margin — the classifier that commits to its decisions with the most room to spare. The kernel trick then makes it nonlinear for free: replace every inner product $x\cdot x'$ with a kernel $k(x,x')=\langle\phi(x),\phi(x')\rangle$ and the linear machine operates in a rich implicit feature space $\phi$ without ever forming it.
Two algorithms are built from scratch in svmkernel.py and validated against scikit-learn:
- Pegasos — the soft-margin linear SVM trained by stochastic sub-gradient descent on the hinge objective (Shalev-Shwartz et al., 2007). The margin idea and the hinge loss are the SVM; this is the optimiser.
- Kernel ridge regression — the closed form $\alpha=(K+\lambda I)^{-1}y$, which solves the same linear system as scikit-learn and, with an RBF kernel, equals the posterior mean of a Gaussian process. That identity is the bridge back to the Bayesian-Nonparametric benchmark: the SVM's regression cousin is a GP.
Same two datasets and running scoreboard (credit default, California housing). The R companion uses e1071 and kernlab. ROC-AUC is defined in the CART notebook.
1. Max-margin and the hinge loss — a linear SVM from scratch¶
The SVM minimises $\frac{\lambda}{2}\lVert w\rVert^2 + \frac1n\sum_i \max(0,\,1-y_i\,w\!\cdot\!x_i)$: the hinge term is zero once a point sits on the correct side of the margin and grows linearly inside it, so only points on or inside the margin — the support vectors — shape the boundary. PegasosSVM optimises it by SGD. The left panel shows the learned max-margin boundary and its support vectors on a 2-D toy; then we validate on the credit data against scikit-learn's LinearSVC.
import numpy as np, pandas as pd, matplotlib.pyplot as plt, time, warnings
warnings.filterwarnings("ignore")
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, mean_squared_error
from svmkernel import PegasosSVM, KernelRidge
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
rng=np.random.default_rng(0)
# 2-D toy: two separable-ish gaussian blobs
n0=120; A=rng.normal([-1.3,-1.3],0.85,(n0,2)); B=rng.normal([1.3,1.3],0.85,(n0,2))
Xt=np.vstack([A,B]); yt=np.r_[np.zeros(n0),np.ones(n0)]
svm=PegasosSVM(lam=5e-3,n_epochs=60).fit(Xt,yt)
w,b=svm.coef_,svm.intercept_; marg=(yt*2-1)*(Xt@w+b) # signed margin y*f(x): <1 is on or inside it
sv=marg<1.0
fig,ax=plt.subplots(1,2,figsize=(12.5,4.8))
xx,yy=np.meshgrid(np.linspace(-4,4,300),np.linspace(-4,4,300)); Z=(np.c_[xx.ravel(),yy.ravel()]@w+b).reshape(xx.shape)
ax[0].contourf(xx,yy,Z>0,alpha=.08,colors=[BLUE,RED]); ax[0].contour(xx,yy,Z,levels=[-1,0,1],colors=["k","k","k"],linestyles=["--","-","--"],linewidths=[1,2,1])
ax[0].scatter(*Xt[yt==0].T,s=16,color=BLUE); ax[0].scatter(*Xt[yt==1].T,s=16,color=RED)
ax[0].scatter(*Xt[sv].T,s=90,facecolors="none",edgecolors=GREEN,linewidths=1.5,label="support vectors")
ax[0].set_title("Max-margin boundary (from-scratch Pegasos)"); ax[0].legend(loc="upper left",fontsize=8); ax[0].set_xlim(-4,4); ax[0].set_ylim(-4,4)
# credit validation
d=pd.read_csv("credit_default.csv"); feat=[c for c in d.columns if c!="default"]
Xc=d[feat].to_numpy(float); yc=d["default"].to_numpy(int)
Xc_tr,Xc_te,yc_tr,yc_te=train_test_split(Xc,yc,test_size=0.3,random_state=0,stratify=yc)
scc=StandardScaler().fit(Xc_tr); Zc_tr=scc.transform(Xc_tr); Zc_te=scc.transform(Xc_te)
from sklearn.svm import LinearSVC
peg=PegasosSVM(lam=1e-4,n_epochs=40).fit(Zc_tr,yc_tr); lsvc=LinearSVC(C=1.0,max_iter=5000).fit(Zc_tr,yc_tr)
svm_lin_auc=roc_auc_score(yc_te,peg.decision_function(Zc_te))
lsvc_auc=roc_auc_score(yc_te,lsvc.decision_function(Zc_te))
from sklearn.linear_model import LogisticRegression
log_auc=roc_auc_score(yc_te,LogisticRegression(max_iter=2000).fit(Zc_tr,yc_tr).predict_proba(Zc_te)[:,1])
ax[1].axis("off"); ax[1].text(0.0,0.5,
f"Credit default — linear SVM (test AUC)\n\n from-scratch Pegasos : {svm_lin_auc:.3f}\n sklearn LinearSVC : {lsvc_auc:.3f}\n\n"
f" logistic baseline : {log_auc:.3f}\n\nThe exact SVM ties the logistic; the SGD\nsolver gives up a little more. Max-margin\nbuys nothing here: the classes overlap\nheavily and the boundary is near-linear.",
fontsize=11,va="center",family="monospace")
plt.tight_layout(); plt.show()
print(f"From-scratch Pegasos ({svm_lin_auc:.3f}) tracks sklearn's LinearSVC ({lsvc_auc:.3f}) to {abs(svm_lin_auc-lsvc_auc):.3f} AUC -- the same")
print(f"hinge objective, SGD against the exact solver. Set beside the logistic baseline ({log_auc:.3f}), the exact SVM is a")
print(f"genuine tie, and the {log_auc-svm_lin_auc:.3f} the from-scratch version trails is the optimiser rather than the loss.")
print(f"Only the {int(sv.sum())} circled points satisfy y*f(x) < 1 -- on or inside the margin. The other {int((~sv).sum())} could be moved")
print("anywhere on their own side of the margin without shifting the boundary by a hair; that is what 'support' means.")
From-scratch Pegasos (0.704) tracks sklearn's LinearSVC (0.711) to 0.007 AUC -- the same hinge objective, SGD against the exact solver. Set beside the logistic baseline (0.715), the exact SVM is a genuine tie, and the 0.011 the from-scratch version trails is the optimiser rather than the loss. Only the 18 circled points satisfy y*f(x) < 1 -- on or inside the margin. The other 222 could be moved anywhere on their own side of the margin without shifting the boundary by a hair; that is what 'support' means.
2. The kernel trick — going nonlinear¶
A linear SVM cannot separate classes that curl around each other. The kernel trick swaps $x\cdot x'$ for $k(x,x')$ — most often the RBF $k(x,x')=e^{-\gamma\lVert x-x'\rVert^2}$ — implicitly lifting the data into an infinite-dimensional space where a linear margin becomes a curved boundary in the original space. On the classic two-moons data the linear SVM fails and the RBF SVM separates cleanly. On the real credit data the RBF kernel lifts the SVM only back to the logistic baseline — an honest reminder that the kernel helps only when there is smooth nonlinear structure to exploit, and heavily-overlapping imbalanced classes offer little.
from sklearn.datasets import make_moons
from sklearn.svm import SVC
Xm,ym=make_moons(n_samples=400,noise=0.25,random_state=0)
lin=SVC(kernel="linear",C=1.0).fit(Xm,ym); rbf=SVC(kernel="rbf",C=1.0,gamma=1.0).fit(Xm,ym)
fig,ax=plt.subplots(1,2,figsize=(12.5,4.6))
xx,yy=np.meshgrid(np.linspace(-2,3,300),np.linspace(-1.5,2,300)); grid=np.c_[xx.ravel(),yy.ravel()]
for a,(mdl,ttl) in zip(ax,[(lin,"Linear SVM — cannot separate moons"),(rbf,"RBF-kernel SVM — curved margin")]):
a.contourf(xx,yy,mdl.predict(grid).reshape(xx.shape),alpha=.12,colors=[BLUE,RED])
a.contour(xx,yy,mdl.decision_function(grid).reshape(xx.shape),levels=[0],colors="k",linewidths=2)
a.scatter(*Xm[ym==0].T,s=12,color=BLUE); a.scatter(*Xm[ym==1].T,s=12,color=RED)
a.set_title(ttl+f" (acc {mdl.score(Xm,ym):.2f})")
plt.tight_layout(); plt.show()
# RBF SVM on credit subsample (SVC is O(n^2) -> subsample, like the GP)
m=2500; s=rng.choice(len(Zc_tr),m,replace=False)
t=time.time(); csvc=SVC(kernel="rbf",C=1.0,gamma=0.02).fit(Zc_tr[s],yc_tr[s]); ct=time.time()-t
svm_rbf_auc=roc_auc_score(yc_te,csvc.decision_function(Zc_te))
print(f"RBF SVM on credit (subsample {m}, O(n^2) like the GP): test AUC {svm_rbf_auc:.3f} fit {ct:.2f}s")
print(f"The kernel lifts the SVM to the logistic baseline ({log_auc:.3f}) and stops there, a shade above the linear SVM")
print(f"({svm_lin_auc:.3f}). On this noisy, 22%-imbalanced data there is little smooth nonlinear structure to find, so the")
print("curved boundary costs O(n^2) and returns nothing. The kernel's payoff shows on the regression task below.")
RBF SVM on credit (subsample 2500, O(n^2) like the GP): test AUC 0.715 fit 0.07s The kernel lifts the SVM to the logistic baseline (0.715) and stops there, a shade above the linear SVM (0.704). On this noisy, 22%-imbalanced data there is little smooth nonlinear structure to find, so the curved boundary costs O(n^2) and returns nothing. The kernel's payoff shows on the regression task below.
3. Kernel ridge regression & SVR — California¶
For regression the kernel attaches to ridge regression: kernel ridge solves $\alpha=(K+\lambda I)^{-1}y$ and predicts $f(x)=\sum_i\alpha_i k(x,x_i)$ — ridge in the implicit feature space. svmkernel.KernelRidge implements exactly this and matches scikit-learn to $10^{-13}$. Its close relative SVR uses an $\varepsilon$-insensitive tube (errors under $\varepsilon$ cost nothing) for a sparse, robust fit. Both are fit on a 1,500-point subsample ($O(n^3)$, as with the GP) and both beat the linear model on California — this is where the kernel earns its keep.
from sklearn.kernel_ridge import KernelRidge as SKKR
from sklearn.svm import SVR
h=pd.read_csv("cali_housing.csv"); hf=[c for c in h.columns if c!="MedHouseVal"]
Xh=h[hf].to_numpy(float); yh=h["MedHouseVal"].to_numpy(float)
Xh_tr,Xh_te,yh_tr,yh_te=train_test_split(Xh,yh,test_size=0.3,random_state=0)
sch=StandardScaler().fit(Xh_tr); Rtr=sch.transform(Xh_tr); Rte=sch.transform(Xh_te)
m2=1500; s2=rng.choice(len(Rtr),m2,replace=False)
ym,ys=yh_tr[s2].mean(),yh_tr[s2].std(); yz=(yh_tr[s2]-ym)/ys # standardize target outside the class
gam=1.0/Rtr.shape[1]
kr=KernelRidge(kernel="rbf",gamma=gam,alpha=1.0).fit(Rtr[s2],yz)
sk=SKKR(kernel="rbf",gamma=gam,alpha=1.0).fit(Rtr[s2],yz)
pk=kr.predict(Rte)*ys+ym; psk=sk.predict(Rte)*ys+ym
kr_rmse=mean_squared_error(yh_te,pk)**.5
svr=SVR(kernel="rbf",C=1.0,gamma="scale").fit(Rtr[s2],yh_tr[s2]); svr_rmse=mean_squared_error(yh_te,svr.predict(Rte))**.5
print(f"KernelRidge from-scratch vs sklearn: max|diff| {np.abs(pk-psk).max():.2e} (identical linear system)")
print(f"California test RMSE -- kernel ridge {kr_rmse:.3f} SVR {svr_rmse:.3f} (linear model 0.737)")
fig,ax=plt.subplots(figsize=(6.2,5)); ax.scatter(pk,yh_te,s=5,alpha=.12,color=BLUE)
qb=np.quantile(pk,np.linspace(0,1,11)); idx=np.clip(np.digitize(pk,qb[1:-1]),0,9)
ax.plot([pk.min(),pk.max()],[pk.min(),pk.max()],"k--",lw=1,label="perfect")
ax.plot([pk[idx==j].mean() for j in range(10)],[yh_te[idx==j].mean() for j in range(10)],"o-",color=RED,lw=2,label="decile means")
ax.set_xlabel("kernel-ridge predicted value ($100k)"); ax.set_ylabel("actual value"); ax.set_title(f"California — kernel ridge (RMSE {kr_rmse:.3f})"); ax.legend()
plt.tight_layout(); plt.show()
print("Kernel ridge/SVR clear the linear model (0.74 -> 0.63) by bending to the nonlinear income-price surface -- the same")
print("gain the trees and the GP got. The decile means still sag at the top: the $500k price CAP no smooth model escapes.")
KernelRidge from-scratch vs sklearn: max|diff| 3.95e-14 (identical linear system) California test RMSE -- kernel ridge 0.647 SVR 0.644 (linear model 0.737)
Kernel ridge/SVR clear the linear model (0.74 -> 0.63) by bending to the nonlinear income-price surface -- the same gain the trees and the GP got. The decile means still sag at the top: the $500k price CAP no smooth model escapes.
4. The bridge — kernel ridge is a Gaussian process¶
The identity promised in the Bayesian-Nonparametric benchmark, made concrete. A GP with kernel $k$ and observation-noise variance $\sigma^2$ has posterior mean $$\bar f(x_*) = k(x_*,X)\,[K+\sigma^2 I]^{-1}y,$$ which is exactly kernel ridge regression with $\lambda=\sigma^2$. So the SVM's regression cousin and the Bayesian nonparametric workhorse compute the same point prediction — they differ only in that the GP also returns the posterior variance (the uncertainty), while kernel ridge returns the mean alone. We verify the predictions coincide to machine precision, and then settle the one thing that identity leaves open: why the GP notebook's RMSE is lower than the kernel-ridge fit above, if the two are the same estimator.
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, ConstantKernel as Cst
gp=GaussianProcessRegressor(kernel=Cst(1.0)*RBF(length_scale=1/np.sqrt(2*gam)), alpha=1.0,
optimizer=None, normalize_y=False).fit(Rtr[s2], yz) # matched kernel + noise
pgp=gp.predict(Rte)*ys+ym
fig,ax=plt.subplots(figsize=(5.6,5)); ax.scatter(pk,pgp,s=6,alpha=.3,color=PURP); lim=[pk.min(),pk.max()]
ax.plot(lim,lim,"k--",lw=1); ax.set_xlabel("kernel ridge prediction"); ax.set_ylabel("GP posterior mean")
ax.set_title(f"Same prediction: max|diff| {np.abs(pk-pgp).max():.1e}"); plt.tight_layout(); plt.show()
print(f"kernel ridge vs GP posterior mean: max|diff| {np.abs(pk-pgp).max():.2e}, correlation {np.corrcoef(pk,pgp)[0,1]:.6f}.")
print("They ARE the same estimator. The GP notebook's regression result (RMSE 0.611) is this same kernel machine, fit with")
print("its length-scale and noise chosen by marginal likelihood and carrying the extra posterior variance for uncertainty.")
from sklearn.model_selection import GridSearchCV
gs_kr=GridSearchCV(SKKR(kernel="rbf"),{"gamma":[0.02,0.05,0.125,0.25,0.5],"alpha":[0.01,0.05,0.1,0.3,1.0]},
cv=5,scoring="neg_root_mean_squared_error").fit(Rtr[s2],yz)
kr_tuned=mean_squared_error(yh_te,gs_kr.predict(Rte)*ys+ym)**.5
print(f"\nThat identity leaves one thing to explain: kernel ridge scored RMSE {kr_rmse:.3f} above, while the GP notebook")
print(f"reports 0.611. Choosing the same two numbers by 5-fold CV on the training subsample alone -- the test set never")
print(f"touched -- gives {gs_kr.best_params_} and test RMSE {kr_tuned:.3f}, the GP's value.")
print("So the gap was never the estimator, which is provably the same one. It is that the GP SELECTED its length-scale")
print("and noise by maximising the marginal likelihood, while kernel ridge had them handed to it. That is what the")
print("Bayesian machinery buys here: not a better predictor, but hyperparameters read off the data rather than guessed,")
print("and a posterior variance on top. Left at gamma=1/p and alpha=1, the same estimator gives up 0.036 RMSE.")
kernel ridge vs GP posterior mean: max|diff| 3.38e-14, correlation 1.000000. They ARE the same estimator. The GP notebook's regression result (RMSE 0.611) is this same kernel machine, fit with its length-scale and noise chosen by marginal likelihood and carrying the extra posterior variance for uncertainty.
That identity leaves one thing to explain: kernel ridge scored RMSE 0.647 above, while the GP notebook
reports 0.611. Choosing the same two numbers by 5-fold CV on the training subsample alone -- the test set never
touched -- gives {'alpha': 0.1, 'gamma': 0.125} and test RMSE 0.612, the GP's value.
So the gap was never the estimator, which is provably the same one. It is that the GP SELECTED its length-scale
and noise by maximising the marginal likelihood, while kernel ridge had them handed to it. That is what the
Bayesian machinery buys here: not a better predictor, but hyperparameters read off the data rather than guessed,
and a posterior variance on top. Left at gamma=1/p and alpha=1, the same estimator gives up 0.036 RMSE.
5. The scoreboard¶
Support-vector and kernel methods placed beside the running field on both tasks (frequentist baselines recomputed on the identical split; GP/GAM from the BNP benchmark, BART from its notebook).
from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
import xgboost as xgb
cl={}; rg={}
cl["logistic / linear"]=roc_auc_score(yc_te,LogisticRegression(max_iter=2000).fit(Zc_tr,yc_tr).predict_proba(Zc_te)[:,1])
rg["logistic / linear"]=mean_squared_error(yh_te,LinearRegression().fit(Rtr,yh_tr).predict(Rte))**.5
cl["linear SVM"]=svm_lin_auc; rg["kernel ridge"]=kr_rmse
rg["kernel ridge (tuned)"]=kr_tuned
cl["RBF SVM"]=svm_rbf_auc; rg["SVR"]=svr_rmse
cl["random forest"]=roc_auc_score(yc_te,RandomForestClassifier(n_estimators=400,min_samples_leaf=5,random_state=0,n_jobs=-1).fit(Xc_tr,yc_tr).predict_proba(Xc_te)[:,1])
rg["random forest"]=mean_squared_error(yh_te,RandomForestRegressor(n_estimators=300,min_samples_leaf=3,random_state=0,n_jobs=-1).fit(Xh_tr,yh_tr).predict(Xh_te))**.5
cl["XGBoost"]=roc_auc_score(yc_te,xgb.XGBClassifier(n_estimators=300,learning_rate=0.05,max_depth=4,verbosity=0,n_jobs=-1).fit(Xc_tr,yc_tr).predict_proba(Xc_te)[:,1])
rg["XGBoost"]=mean_squared_error(yh_te,xgb.XGBRegressor(n_estimators=300,learning_rate=0.05,max_depth=4,verbosity=0,n_jobs=-1).fit(Xh_tr,yh_tr).predict(Xh_te))**.5
cl["GP (BNP nb)"]=0.742; rg["GP (BNP nb)"]=0.611
cl["GAM (BNP nb)"]=0.765; rg["GAM (BNP nb)"]=0.643
sc=pd.DataFrame({"credit AUC (higher=better)":cl,"California RMSE (lower=better)":rg})
print(sc.round(3).astype(object).where(sc.notna()," --").to_string()) # -- = method not used for that task
ker={"linear SVM","RBF SVM","kernel ridge","kernel ridge (tuned)","SVR"}
def cc(nm): return [ORANGE if n in ker else (PURP if "BNP" in n else GREY) for n in nm]
fig,ax=plt.subplots(1,2,figsize=(14,4.8))
na=list(cl); ax[0].barh(na,[cl[k] for k in na],color=cc(na)); ax[0].set_xlim(0.5,0.8); ax[0].invert_yaxis(); ax[0].set_title("Credit — test AUC (higher better)")
nb=list(rg); ax[1].barh(nb,[rg[k] for k in nb],color=cc(nb)); ax[1].invert_yaxis(); ax[1].set_title("California — test RMSE (lower better)")
plt.tight_layout(); plt.show()
print("Kernel methods (orange): on credit the linear SVM lands just under the logistic and the RBF kernel just level with")
print(f"it -- the kernel machine does not beat a linear model on linear, overlapping data. On California it does: {kr_rmse:.3f}")
print(f"at assumed hyperparameters and {kr_tuned:.3f} tuned, against {rg['logistic / linear']:.3f} for the linear model, and that tuned figure")
print("is the GP's 0.611, as it must be. Trees and boosting still lead raw accuracy at full scale, and kernels, like the")
print("GP, pay an O(n^2-n^3) cost that forces subsampling -- the recurring reason they do not own large tabular data.")
credit AUC (higher=better) California RMSE (lower=better) logistic / linear 0.715 0.737 linear SVM 0.704 -- RBF SVM 0.715 -- random forest 0.775 0.523 XGBoost 0.774 0.494 GP (BNP nb) 0.742 0.611 GAM (BNP nb) 0.765 0.643 kernel ridge -- 0.647 kernel ridge (tuned) -- 0.612 SVR -- 0.644
Kernel methods (orange): on credit the linear SVM lands just under the logistic and the RBF kernel just level with it -- the kernel machine does not beat a linear model on linear, overlapping data. On California it does: 0.647 at assumed hyperparameters and 0.612 tuned, against 0.737 for the linear model, and that tuned figure is the GP's 0.611, as it must be. Trees and boosting still lead raw accuracy at full scale, and kernels, like the GP, pay an O(n^2-n^3) cost that forces subsampling -- the recurring reason they do not own large tabular data.
Do the scores mean anything as probabilities?¶
Everything above is AUC, which only judges ranking. That matters more than usual for a support vector machine, because an SVM does not produce a probability at all — its natural output is a signed distance from the hyperplane. To get a probability out you have to bolt one on, which scikit-learn and e1071 both do by Platt scaling: fit a logistic regression to the decision values on held-out folds.
So the question is whether that retro-fitted probability is honest. Below, the RBF SVM with Platt scaling is binned by predicted probability and compared to the observed default rate, against a logistic regression that estimates a probability directly.
from sklearn.svm import SVC as _SVC
_pl = _SVC(kernel="rbf", C=1.0, gamma=0.02, probability=True, random_state=0).fit(Zc_tr[s], yc_tr[s])
_p_svm = _pl.predict_proba(Zc_te)[:,1]
_p_log = LogisticRegression(max_iter=2000).fit(Zc_tr, yc_tr).predict_proba(Zc_te)[:,1]
def _rel(p, y, nb=10):
q = np.quantile(p, np.linspace(0,1,nb+1)); b = np.clip(np.digitize(p, q[1:-1]), 0, nb-1)
pp = [p[b==k].mean() for k in range(nb)]; oo = [y[b==k].mean() for k in range(nb)]
return pp, oo, float(np.sum([np.mean(b==k)*abs(oo[k]-pp[k]) for k in range(nb)]))
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
for _p,_c,_n in [(_p_svm,ORANGE,"RBF SVM + Platt"),(_p_log,GREY,"logistic")]:
pp,oo,e = _rel(_p, yc_te)
ax[0].plot(pp,oo,"o-",color=_c,lw=2,label=f"{_n} (ECE {e:.3f})")
ax[1].hist(_p,bins=40,histtype="step",lw=2,color=_c,label=_n)
print(f"{_n:>18} AUC {roc_auc_score(yc_te,_p):.3f} ECE {e:.3f} mean pred {_p.mean():.3f} "
f"base rate {yc_te.mean():.3f} max gap {max(abs(np.array(oo)-np.array(pp))):.3f}")
_mx=max(_p_svm.max(),_p_log.max())
ax[0].plot([0,_mx],[0,_mx],"k--",lw=1,label="perfect"); ax[0].set_xlabel("predicted P(default)")
ax[0].set_ylabel("observed default rate"); ax[0].set_title("Is a Platt-scaled SVM score a probability?"); ax[0].legend(fontsize=8)
ax[1].set_xlabel("predicted P(default)"); ax[1].set_ylabel("clients"); ax[1].set_yscale("log")
ax[1].set_title("Distribution of predicted probabilities"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("\nThe SVM never estimated a probability; Platt scaling fitted one to its decision values after the fact, and")
print("the reliability curve is the test of whether that worked. Read the ECE column against the AUC column: they")
print("are different properties, and a margin-based classifier can rank well while its retro-fitted probabilities")
print("drift. Where the two disagree, use the ranking for triage and the calibrated model for anything that feeds")
print("an expected-loss calculation -- the same conclusion the gradient-boosting notebook reaches from the other side.")
RBF SVM + Platt AUC 0.715 ECE 0.030 mean pred 0.220 base rate 0.221 max gap 0.083
logistic AUC 0.715 ECE 0.059 mean pred 0.219 base rate 0.221 max gap 0.110
The SVM never estimated a probability; Platt scaling fitted one to its decision values after the fact, and the reliability curve is the test of whether that worked. Read the ECE column against the AUC column: they are different properties, and a margin-based classifier can rank well while its retro-fitted probabilities drift. Where the two disagree, use the ranking for triage and the calibrated model for anything that feeds an expected-loss calculation -- the same conclusion the gradient-boosting notebook reaches from the other side.
6. Summary¶
A support vector machine is a max-margin classifier under the hinge loss; the kernel trick makes it — and its ridge-regression cousin — nonlinear by swapping inner products for a kernel. From scratch: PegasosSVM reproduced scikit-learn's linear SVM to ~0.01 AUC, and KernelRidge matched it to $10^{-13}$. The kernel trick separated the two-moons data a linear machine could not; on the real data the RBF kernel added little to credit classification (overlapping, imbalanced) but let kernel ridge and SVR beat the linear model on California (0.737 → 0.645), the same nonlinear gain the trees and GP achieved.
The unifying result is the bridge in §4: kernel ridge regression is the posterior mean of a Gaussian process, verified to machine precision — and the residual RMSE gap to the GP notebook turns out to be hyperparameter selection, not estimation: tuned by cross-validation the same kernel ridge lands on the GP's number. What the marginal likelihood buys is the choice of length-scale and noise, plus a posterior variance. It ties this subsection to the Bayesian-Nonparametric benchmark — the frequentist kernel machine and the Bayesian one are the same estimator, differing only in that the GP also reports uncertainty. And kernels share the GP's $O(n^2\!-\!n^3)$ scaling, the recurring reason gradient boosting, not the kernel machine, is the default for large tabular data.
That closes the core of the Regularized & Kernel Learning subsection: penalised linear models (Ridge/Lasso/Elastic Net) and their kernelised, max-margin relatives (SVM/SVR/kernel ridge), with the Bayesian counterparts — shrinkage priors and Gaussian processes — cross-linked throughout. The R companion fits the same models with e1071 and kernlab.