ML Arc, Foundations — SVM and the Discriminative Spectrum

The support vector machine models only the boundary — completing the generative→discriminative picture

The previous notebook contrasted generative classifiers (Naive Bayes, and linear discriminant analysis or LDA — model $p(x\mid y)p(y)$) with a discriminative one (logistic regression — model $p(y\mid x)$). The support vector machine completes the spectrum by going one step further in the discriminative direction: it models even less. Logistic regression still estimates a full conditional probability $p(y\mid x)$; the SVM estimates only the decision boundary, placing it to maximize the margin between the classes, and cares only about the points nearest that boundary — the support vectors. It makes no probability model at all.

That gives a clean ladder of "how much of the data-generating process you commit to":

	models	commits to	gives you
Generative (NB, LDA)	$p(x\mid y)\,p(y)$ — everything	a full model of the features	probabilities (often miscalibrated) + fast convergence
Probabilistic-discriminative (logistic)	$p(y\mid x)$	the conditional distribution	probabilities natively — though not necessarily good ones
Geometric-discriminative (SVM)	just the boundary (max margin)	nothing about the distribution	a robust boundary — but no probabilities

Modeling less means fewer assumptions to get wrong (robustness to misspecification and outliers) but also less output — the SVM's raw scores are not probabilities and must be calibrated (Platt scaling) before they can be used as such, a different calibration failure than Naive Bayes'. And the kernel trick lets the SVM draw nonlinear boundaries without ever modeling $p(x)$. We build the intuition in 2-D, then compare all three on the credit data. Python-lead (scikit-learn; the from-scratch max-margin/Pegasos and kernel machinery live in the SVM and Kernel Methods notebook); R companion uses e1071.

1. Maximum margin — the SVM only cares about the boundary

On a 2-D two-class problem, the SVM finds the separating line with the widest margin — the largest empty corridor between the classes — defined entirely by a handful of support vectors (the points on the margin's edge). Every point far from the boundary is irrelevant to the fit: move or delete it and nothing changes. This is the opposite of a generative classifier, which uses the entire distribution of each class. Logistic regression sits in between — it uses all points but weights them by how close they are to the boundary. The plot shows the SVM's max-margin boundary and support vectors against the logistic boundary; the two often differ because they optimize different things (margin vs conditional likelihood).

import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
rng=np.random.default_rng(1)
X=np.vstack([rng.normal([-1.3,-1.0],0.9,(60,2)), rng.normal([1.6,1.4],0.9,(60,2))]); y=np.r_[np.zeros(60),np.ones(60)]
svm=SVC(kernel="linear",C=1).fit(X,y); lg=LogisticRegression().fit(X,y)
xx,yy=np.meshgrid(np.linspace(X[:,0].min()-1,X[:,0].max()+1,300),np.linspace(X[:,1].min()-1,X[:,1].max()+1,300))
Z=svm.decision_function(np.c_[xx.ravel(),yy.ravel()]).reshape(xx.shape)
fig,ax=plt.subplots(figsize=(8,5.5))
ax.scatter(X[y==0,0],X[y==0,1],c=BLUE,s=25,label="class 0"); ax.scatter(X[y==1,0],X[y==1,1],c=RED,s=25,label="class 1")
ax.contour(xx,yy,Z,levels=[-1,0,1],colors=[GREY,"k",GREY],linestyles=["--","-","--"],linewidths=[1,2,1])
sv=svm.support_vectors_; ax.scatter(sv[:,0],sv[:,1],s=140,facecolors="none",edgecolors=GREEN,linewidths=2,label="support vectors")
lgb=-(lg.coef_[0,0]*xx[0]+lg.intercept_[0])/lg.coef_[0,1]; ax.plot(xx[0],lgb,color=PURP,lw=2,ls=":",label="logistic boundary")
ax.set_xlim(xx.min(),xx.max()); ax.set_ylim(yy.min(),yy.max()); ax.set_title("SVM max-margin boundary (solid) + margins (dashed) + support vectors"); ax.legend(fontsize=8)
plt.tight_layout(); plt.show()
print(f"The SVM boundary is fixed by only {len(sv)} support vectors of {len(X)} points -- {100*len(sv)/len(X):.0f}% -- and the rest could be")
print("deleted with no effect. Hold onto that number: section 4 refits the same idea on real, overlapping data, where")
print("it looks very different, because this sparsity is a property of a SEPARABLE problem rather than of the method.")
print("A generative classifier uses the whole class distribution; the SVM uses only the margin. Logistic (dotted) differs")
print("because it maximizes conditional likelihood, not margin -- three different objectives, three (slightly) different lines.")

The SVM boundary is fixed by only 5 support vectors of 120 points -- 4% -- and the rest could be
deleted with no effect. Hold onto that number: section 4 refits the same idea on real, overlapping data, where
it looks very different, because this sparsity is a property of a SEPARABLE problem rather than of the method.
A generative classifier uses the whole class distribution; the SVM uses only the margin. Logistic (dotted) differs
because it maximizes conditional likelihood, not margin -- three different objectives, three (slightly) different lines.

2. The kernel trick — nonlinear boundaries without modeling $p(x)$

When classes are not linearly separable, the SVM applies the kernel trick: it implicitly maps the features into a high-dimensional space and finds a linear margin there, which is a curved boundary back in the original space — all without ever writing down a density for $x$. The RBF (Gaussian) kernel is the default, turning the SVM into a flexible nonlinear classifier whose complexity is controlled by the margin (regularization $C$) and kernel width $\gamma$. On a two-ring problem that no linear classifier can solve, the RBF-SVM traces the correct circular boundary. (The kernel machinery — and the striking identity kernel ridge regression = Gaussian-process posterior mean — is developed from scratch in the SVM and Kernel Methods notebook; here it completes the discriminative picture.)

from sklearn.datasets import make_circles
Xc,yc=make_circles(300,noise=0.12,factor=0.4,random_state=0)
fig,ax=plt.subplots(1,2,figsize=(13,4.8))
for a,(k,ttl) in zip(ax,[("linear","linear SVM (fails — no straight line separates rings)"),("rbf","RBF-kernel SVM (nonlinear margin)")]):
    m=SVC(kernel=k,C=2,gamma="scale").fit(Xc,yc)
    xx,yy=np.meshgrid(np.linspace(-1.6,1.6,300),np.linspace(-1.6,1.6,300))
    Z=m.decision_function(np.c_[xx.ravel(),yy.ravel()]).reshape(xx.shape)
    a.contourf(xx,yy,Z>0,alpha=.12,colors=[BLUE,RED]); a.contour(xx,yy,Z,levels=[0],colors="k",linewidths=2)
    a.scatter(Xc[yc==0,0],Xc[yc==0,1],c=BLUE,s=15); a.scatter(Xc[yc==1,0],Xc[yc==1,1],c=RED,s=15)
    a.set_title(f"{ttl}\n(train acc {m.score(Xc,yc):.2f})",fontsize=9); a.set_xticks([]); a.set_yticks([])
plt.tight_layout(); plt.show()
print("The RBF kernel lets the SVM separate the rings with a curved boundary a linear model cannot draw -- flexible")
print("discriminative learning with no model of the feature distribution. Complexity is tuned by C and gamma, not by a density.")

The RBF kernel lets the SVM separate the rings with a curved boundary a linear model cannot draw -- flexible
discriminative learning with no model of the feature distribution. Complexity is tuned by C and gamma, not by a density.

3. The SVM has no probabilities — the other end of the calibration story

Because it models only a boundary, the SVM's natural output is a signed distance (the decision function), not a probability — it is unbounded and has no probabilistic meaning. To get $p(y\mid x)$ you must calibrate the scores, typically with Platt scaling (fit a logistic curve to the decision values on held-out data). This is the mirror image of Naive Bayes' calibration failure from the previous notebook:

• Naive Bayes has probabilities but they are wrong (overconfident, pinned to 0/1 by the independence assumption) — recalibration fixes badly-distorted numbers.
• The SVM has no probabilities at all — calibration creates them from the raw scores.
• Logistic regression produces probabilities natively, with no extra step.

The tempting third line is that logistic therefore gives you the right ones, and it is worth resisting, because this collection has already tested it. The Calibration example found logistic to be the second-worst calibrated of four models on this very dataset — more than four times its own noise floor, and improved eight-fold by isotonic recalibration. "Natively" is not the same as "well", and the measurement below makes the point sharply: a Platt-scaled SVM comes out better calibrated than logistic is on its own.

So the ladder is a probability ladder in a narrower sense than the slogan suggests. What separates the three is whether a probability exists and where it comes from — supplied by a wrong model of $x$, supplied by a fitted link, or supplied by an explicit calibration step — not a clean ordering of who is best calibrated.

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
d=pd.read_csv("credit_default.csv"); feat=[c for c in d.columns if c!="default"]
X2=d[feat].values.astype(float); y2=d["default"].values
Xtr,Xte,ytr,yte=train_test_split(X2,y2,test_size=0.3,random_state=0,stratify=y2)
sc=StandardScaler().fit(Xtr); Ztr=sc.transform(Xtr); Zte=sc.transform(Xte)
sub=np.random.default_rng(0).choice(len(Ztr),4000,replace=False)
svm_raw=SVC(kernel="rbf",C=1,gamma="scale").fit(Ztr[sub],ytr[sub])          # no probabilities
svm_platt=SVC(kernel="rbf",C=1,gamma="scale",probability=True).fit(Ztr[sub],ytr[sub])  # Platt-calibrated
dv=svm_raw.decision_function(Zte); pp=svm_platt.predict_proba(Zte)[:,1]
def ece(p,y,nb=10):
    b=np.linspace(0,1,nb+1);e=0
    for i in range(nb):
        m=(p>=b[i])&(p<b[i+1] if i<nb-1 else p<=b[i+1])
        if m.sum(): e+=m.mean()*abs(y[m].mean()-p[m].mean())
    return e
def reliability(p,yv,nb=10):
    b=np.linspace(0,1,nb+1);xs=[];ys=[]
    for i in range(nb):
        m=(p>=b[i])&(p<b[i+1] if i<nb-1 else p<=b[i+1])
        if m.sum()>30: xs.append(p[m].mean()); ys.append(yv[m].mean())
    return np.array(xs),np.array(ys)
print(f"SVM raw decision values: range [{dv.min():.2f}, {dv.max():.2f}] -- NOT probabilities.")
print(f"After Platt scaling: probabilities in [0,1], ECE = {ece(pp,yte):.3f} (well-calibrated).")
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
ax[0].hist(dv,bins=40,color=ORANGE,alpha=.8); ax[0].axvline(0,color="k",ls="--"); ax[0].set_xlabel("SVM decision value (signed distance)"); ax[0].set_ylabel("count"); ax[0].set_title("Raw SVM scores are unbounded, not probabilities")
ax[1].plot([0,1],[0,1],"k--",lw=1,label="perfect"); xs,ys=reliability(pp,yte); ax[1].plot(xs,ys,"o-",color=GREEN,lw=2,label=f"SVM + Platt (ECE {ece(pp,yte):.2f})")
ax[1].set_xlabel("mean predicted probability"); ax[1].set_ylabel("observed default frequency"); ax[1].set_title("Platt scaling turns SVM scores into calibrated probabilities"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("The SVM commits to nothing probabilistic, so it gives a distance, not a probability -- Platt scaling supplies the")
print("probability model the SVM declined to assume. Naive Bayes had wrong probabilities; the SVM has none until calibrated.")

SVM raw decision values: range [-2.04, 2.12] -- NOT probabilities.
After Platt scaling: probabilities in [0,1], ECE = 0.017 (well-calibrated).

The SVM commits to nothing probabilistic, so it gives a distance, not a probability -- Platt scaling supplies the
probability model the SVM declined to assume. Naive Bayes had wrong probabilities; the SVM has none until calibrated.

4. The three classifiers on the credit data

Finally we line up a generative (Naive Bayes), a probabilistic-discriminative (logistic), and a geometric-discriminative (SVM, linear and RBF) classifier on the credit-default problem.

One thing has to be equalised first. An RBF kernel is $O(n^2)$ in the training size, so it is routine to fit the SVM on a subsample while the linear models see everything — and that quietly turns a comparison of paradigms into a comparison of sample sizes. Both versions are therefore run below: every model on the same 4,000-row subsample, and then the models that scale cheaply on the full 21,000. The ranking survives the correction, which is worth knowing rather than assuming.

The headline is that on this dataset the paradigm matters less for discrimination than for what else each hands you — though "close" should be stated with the actual numbers rather than a rounded range, since the linear SVM is a clear step behind the rest.

from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score
# every model on the SAME 4000-row subsample -- paradigm compared to paradigm
specs=[("Naive Bayes (generative)",lambda: GaussianNB(),"yes (miscalibrated)"),
       ("Logistic (prob. discriminative)",lambda: LogisticRegression(max_iter=2000),"yes, natively"),
       ("Linear SVM (geometric)",lambda: SVC(kernel="linear",C=1,probability=True),"no (Platt supplies them)"),
       ("RBF SVM (geometric+kernel)",lambda: SVC(kernel="rbf",C=1,gamma="scale",probability=True),"no (Platt supplies them)")]
rows=[]
for nm,mk,pr in specs:
    m=mk().fit(Ztr[sub],ytr[sub]); p=m.predict_proba(Zte)[:,1]
    rows.append((nm,roc_auc_score(yte,p),accuracy_score(yte,m.predict(Zte)),ece(p,yte),pr))
tab=pd.DataFrame(rows,columns=["classifier","AUC","accuracy","ECE","native probabilities?"]).set_index("classifier")
print(f"All four on the SAME {len(sub)} training rows:")
print(tab.round(3).to_string())

# and the cheap models on everything, to show the subsample is not what drives the ranking
print(f"\nThe models that scale cheaply, on the full {len(Ztr)} training rows:")
for nm,mk,_ in specs[:3]:
    m=mk().fit(Ztr,ytr); p=m.predict_proba(Zte)[:,1]
    print(f"  {nm:34s} AUC {roc_auc_score(yte,p):.4f}   accuracy {accuracy_score(yte,m.predict(Zte)):.4f}   ECE {ece(p,yte):.4f}")
print(f"  (the RBF SVM is left out here: an O(n^2) kernel on {len(Ztr)} rows takes minutes, which is itself part of the")
print("   paradigm's cost and the reason the subsample was there in the first place.)")
print(f"\nThe ranking is the same either way, so the subsample was not driving it. Two details are worth reading off:")
print(f"Naive Bayes scores HIGHER on the smaller sample ({tab.loc['Naive Bayes (generative)','AUC']:.4f} against {roc_auc_score(yte,GaussianNB().fit(Ztr,ytr).predict_proba(Zte)[:,1]):.4f} on the full set) while")
print("logistic goes the other way -- the previous notebook's convergence result showing up again, unprompted.")
fig,ax=plt.subplots(figsize=(8.5,4))
ax.barh(tab.index,tab["AUC"],color=[GREEN,BLUE,ORANGE,RED]); ax.set_xlim(0.6,0.75); ax.invert_yaxis()
for i,v in enumerate(tab["AUC"]): ax.text(v+0.002,i,f"{v:.3f}",va="center",fontsize=9)
ax.set_xlabel("test AUC"); ax.set_title("Credit: generative, probabilistic-discriminative, and geometric-discriminative — close on ranking")
plt.tight_layout(); plt.show()
_a=tab["AUC"]
print(f"\nOn ranking the four span {_a.min():.3f} to {_a.max():.3f}. That is close, but not uniformly so: the linear SVM at {_a['Linear SVM (geometric)']:.3f} is a")
print(f"clear step behind, and the kernel is what recovers it ({_a['RBF SVM (geometric+kernel)']:.3f}). Naive Bayes ranks best of all -- and classifies worst,")
print(f"at {tab.loc['Naive Bayes (generative)','accuracy']:.3f} accuracy against a do-nothing baseline of {max(yte.mean(),1-yte.mean()):.3f}, which is the miscalibration of the previous notebook.")
print(f"\nOn calibration the ladder does not order the way the slogan does. The Platt-scaled RBF SVM reaches ECE {tab.loc['RBF SVM (geometric+kernel)','ECE']:.4f},")
print(f"better than logistic's native {tab.loc['Logistic (prob. discriminative)','ECE']:.4f} -- an explicit calibration step beats a fitted link here. The Calibration")
print("notebook reached the same conclusion from the other direction, finding logistic second-worst of four on this data.")
print(f"\nAnd the sparsity promised in section 1 largely evaporates: the RBF SVM keeps {svm_raw.support_.size} support vectors, {100*svm_raw.support_.size/len(sub):.0f}% of the")
print(f"training subsample, against {100*5/120:.0f}% on the separable toy problem. 'Only the points near the boundary matter' is true;")
print("it is just that when classes overlap heavily, almost every point IS near the boundary.")

All four on the SAME 4000 training rows:
                                   AUC  accuracy    ECE     native probabilities?
classifier                                                                       
Naive Bayes (generative)         0.726     0.444  0.444       yes (miscalibrated)
Logistic (prob. discriminative)  0.712     0.807  0.054             yes, natively
Linear SVM (geometric)           0.695     0.810  0.068  no (Platt supplies them)
RBF SVM (geometric+kernel)       0.714     0.817  0.017  no (Platt supplies them)

The models that scale cheaply, on the full 21000 training rows:
  Naive Bayes (generative)           AUC 0.7190   accuracy 0.5194   ECE 0.3781
  Logistic (prob. discriminative)    AUC 0.7145   accuracy 0.8111   ECE 0.0551

  Linear SVM (geometric)             AUC 0.6964   accuracy 0.8110   ECE 0.0641
  (the RBF SVM is left out here: an O(n^2) kernel on 21000 rows takes minutes, which is itself part of the
   paradigm's cost and the reason the subsample was there in the first place.)

The ranking is the same either way, so the subsample was not driving it. Two details are worth reading off:
Naive Bayes scores HIGHER on the smaller sample (0.7257 against 0.7190 on the full set) while
logistic goes the other way -- the previous notebook's convergence result showing up again, unprompted.

On ranking the four span 0.695 to 0.726. That is close, but not uniformly so: the linear SVM at 0.695 is a
clear step behind, and the kernel is what recovers it (0.714). Naive Bayes ranks best of all -- and classifies worst,
at 0.444 accuracy against a do-nothing baseline of 0.779, which is the miscalibration of the previous notebook.

On calibration the ladder does not order the way the slogan does. The Platt-scaled RBF SVM reaches ECE 0.0171,
better than logistic's native 0.0537 -- an explicit calibration step beats a fitted link here. The Calibration
notebook reached the same conclusion from the other direction, finding logistic second-worst of four on this data.

And the sparsity promised in section 1 largely evaporates: the RBF SVM keeps 1839 support vectors, 46% of the
training subsample, against 4% on the separable toy problem. 'Only the points near the boundary matter' is true;
it is just that when classes overlap heavily, almost every point IS near the boundary.

5. Summary

The support vector machine is the maximally discriminative classifier: it models only the decision boundary, chosen to maximize the margin, and depends only on the support vectors near it — no model of $p(x)$, and via the kernel trick, nonlinear boundaries without one. That completes the ladder this foundations section set out:

• Generative (Naive Bayes, LDA) — model everything, $p(x\mid y)p(y)$; cheapest probabilities (often miscalibrated), fastest convergence, most assumptions to get wrong;
• Probabilistic-discriminative (logistic) — model $p(y\mid x)$; probabilities without an extra step, fewer assumptions;
• Geometric-discriminative (SVM) — model only the boundary; most robust to misspecification and outliers, but no native probabilities (Platt scaling supplies them), and more data-hungry.

The ladder is also a calibration story, though not the tidy one usually told. Naive Bayes has probabilities that are wrong; the SVM has none until Platt scaling supplies them; logistic has them natively. What does not follow is that logistic's are therefore the best: measured on the same data, the Platt-scaled SVM reached ECE 0.017 against logistic's native 0.054, and the Calibration example separately found logistic to be the second-worst calibrated of four models here. An explicit calibration step beat a fitted link. The three differ in where a probability comes from, not in a clean ordering of who is best.

On ranking the four spanned 0.695 to 0.726 — close, but not uniformly: the linear SVM is a clear step behind and the kernel is what recovers it. Naive Bayes ranked best of all and classified worst, at 0.444 accuracy against a do-nothing baseline of 0.779, which is the previous notebook's miscalibration showing up again.

One caveat on the comparison itself: an RBF kernel is $O(n^2)$, so it is routine to fit the SVM on a subsample while the linear models see everything — which quietly compares sample sizes rather than paradigms. Running every model on the same 4,000 rows, and then the cheap ones on all 21,000, leaves the ranking unchanged. And the sparsity promised by the toy example largely evaporates: 4% of points were support vectors on the separable 2-D problem, 46% on the overlapping credit data. "Only the points near the boundary matter" is true; when classes overlap, almost every point is near the boundary.

Placement and cross-links. With Naive Bayes/LDA (previous notebook) and the SVM here, the generative→discriminative spectrum is complete, and it is the right opening for the ML arc: it frames every classifier that follows by how much of the data it models. The SVM's kernel machinery and the kernel-ridge = Gaussian-process identity are developed from scratch in Regularized & Kernel Learning – SVM and Kernel Methods; the Platt/calibration thread runs into Model Evaluation – Calibration and Conformal Prediction; and the margin/regularization idea connects to the Ridge/Lasso notebook. Guidance: SVM (with an RBF kernel) for clean-margin, moderate-size, high-dimensional problems where a robust boundary matters and you can calibrate afterwards; logistic when you need probabilities out of the box; a generative classifier when data are scarce.