SVM and the Discriminative Spectrum

Python · scikit-learn · R · e1071

Modelling Only the Boundary

The support vector machine completes the spectrum by modelling less than anything before it. Logistic regression still estimates a full conditional probability; the SVM estimates only the decision boundary, placed to maximise the margin, and depends only on the points nearest it — the support vectors. It writes down no probability model at all, and via the kernel trick it draws nonlinear boundaries without ever modelling $p(x)$ .

What the SVM actually solves

The margin is the width of the empty corridor a boundary can carve between the classes. For a linear rule $f(x)=w^\top x+b$ that width is $2/\lVert w\rVert$ , so widening the corridor means shrinking the weights — the maximum-margin problem is a minimisation of $\lVert w\rVert^2$ subject to every point sitting on the correct side by a full unit of margin. Real classes overlap, so the hard constraint is relaxed with slack variables and a cost $C$ controlling how much violation is tolerated. Small $C$ means a wide, forgiving corridor; large $C$ insists on separating the training data and overfits.

\min_{w,b,\xi}\ \tfrac12\lVert w\rVert^{2} + C\sum_{i} \xi_i \qquad \text{s.t.}\quad y_i\bigl(w^\top x_i + b\bigr) \ge 1-\xi_i,\quad \xi_i \ge 0

Written as a loss, that soft-margin problem is hinge loss plus an $L_2$ penalty — which is exactly the shape of a regularised regression, and the reason the margin is best understood as a regularisation device rather than a geometric curiosity. The hinge is what makes the model sparse: it is flat at zero for any point already correct by a full margin, so those points contribute no gradient and no influence. Only the points with $y_i f(x_i)\le 1$ — on or inside the margin — enter the solution at all. Those are the support vectors.

\underbrace{\sum_i \max\bigl(0,\ 1 - y_i f(x_i)\bigr)}_{\text{hinge: zero once correct by a margin}} + \ \lambda\lVert w\rVert^{2} \qquad\text{vs}\qquad \underbrace{\sum_i \log\bigl(1+e^{-y_i f(x_i)}\bigr)}_{\text{log loss: a likelihood, hence probabilities}}

The hinge is also why there are no probabilities. Logistic regression minimises log loss, which is the negative log-likelihood of a Bernoulli model, so its fitted values are probability estimates by construction. The hinge is not a likelihood for anything — it is a penalty for being on the wrong side — so the minimiser has no probabilistic reading. What comes out is a signed distance from the boundary, unbounded in both directions. And because the solution depends on the data only through inner products, those can be replaced by a kernel $K(x,x')$ , which fits a linear margin in an implicit high-dimensional space and draws a curved boundary back in the original one — flexibility with still no model of $p(x)$ .

The ladder it completes

That gives a ladder of how much of the data-generating process each method commits to: generative models everything, probabilistic-discriminative models the conditional, geometric-discriminative models only the boundary. Fewer commitments mean fewer assumptions to get wrong — and less output.

	models	commits to	gives you
Generative (Naive Bayes, linear discriminant analysis)	p(x\|y) p(y) — everything	a full model of the features	probabilities (often wrong) + fast convergence
Probabilistic-discriminative (logistic)	p(y\|x)	the conditional distribution	probabilities natively — not necessarily good ones
Geometric-discriminative (SVM)	just the boundary	nothing about the distribution	a robust boundary, and no probabilities

The sparsity claim needs a caveat the toy example cannot supply. On a well-separated 2-D problem the boundary is fixed by 5 support vectors out of 120 — 4% — and the rest could be deleted with no effect. On the overlapping credit data the same method keeps 46% of the training subsample. "Only the points near the boundary matter" is true; when classes overlap heavily, almost every point is near the boundary.

The Calibration Ladder Inverts

Because it models only a boundary, the SVM's natural output is a signed distance, not a probability. Platt scaling — fitting a logistic curve to the decision values — supplies the probability model the SVM declined to assume. That is the mirror image of the previous example's failure: Naive Bayes has probabilities that are wrong; the SVM has none until calibration creates them.

The tempting third rung is that logistic therefore gives you the right ones, and it does not survive measurement. On the same data the Platt-scaled SVM reaches an expected calibration error (ECE, the average gap between a stated probability and the frequency observed at it) of 0.017 against logistic's native 0.054 — an explicit calibration step beating a fitted link — and Calibration separately found logistic to be the second-worst calibrated of four models here, improved eight-fold by isotonic recalibration. "Native" is not the same as "good." The three rungs differ in where a probability comes from, not in a clean ordering of who is best calibrated.

In the table, AUC is the area under the receiver-operating-characteristic curve and scores ranking alone; ECE is the expected calibration error, the average gap between a stated probability and the frequency observed at it.

same data, same training rows	AUC (ranking)	accuracy	ECE (calibration)	probabilities?
Naive Bayes (generative)	0.726	0.444	0.444	yes, wrong
Logistic (prob. discriminative)	0.712	0.807	0.054	yes, natively
Linear SVM (geometric)	0.695	0.810	0.068	no — Platt supplies them
RBF SVM (geometric + kernel)	0.714	0.817	0.017	no — Platt supplies them

One thing had to be equalised before the comparison meant anything. An RBF kernel is $O(n^2)$ in the training size, so it is routine to fit the SVM on a subsample while the linear models see everything — which quietly turns a comparison of paradigms into a comparison of sample sizes. Running every model on the same 4,000 rows, and then the cheap ones on all 21,000, leaves the ranking unchanged. The shortcut was not driving it, which is worth knowing rather than assuming.

On ranking the four span 0.695 to 0.726 — close, but not uniformly: the linear SVM is a clear step behind and the kernel is what recovers it. Naive Bayes ranks best of all and classifies worst, which is the previous example's miscalibration appearing again. And a detail that arrives unprompted: Naive Bayes scores higher on the smaller sample (0.726 against 0.719) while logistic goes the other way — the Ng–Jordan convergence result showing up without being asked for.

Where this sits

The generative end of the ladder is Naive Bayes vs Logistic. The kernel machinery — and the identity that kernel ridge regression equals the Gaussian-process posterior mean — is built from scratch in SVM and Kernel Methods, with the Bayesian view in Gaussian Processes & Splines. The margin-as-regularisation idea connects to Ridge, Lasso & Elastic Net, and the Platt thread runs into Calibration and Conformal Prediction, which drops the probability model entirely in favour of a coverage guarantee.

Notebooks

Python The max-margin boundary and support vectors in 2-D, the kernel trick on a two-ring problem, Platt scaling shown to supply probabilities the SVM declines to assume, and a three-paradigm comparison equalised for training size with the calibration ladder found to invert (Python) Open → R e1071 reproducing the same ladder, with the subsample confound corrected and the same calibration inversion confirmed independently — Platt-scaled SVM at 0.017 against logistic native at 0.050 (R) Open →

Downloads

credit_default.csv Default of credit-card clients — 30,000 clients, 23 features, 22.1% default rate (Yeh & Lien 2009; UCI). A credit limit, four demographics, and eighteen months of payment history: six of repayment status, six of bill amounts, six of payments made. Shared by both examples in this section, and by the tree-ensemble, calibration and interpretability sections

References

Cortes, C. & Vapnik, V. (1995). Support-vector networks. Machine Learning 20(3), 273–297.
Boser, B. E., Guyon, I. M. & Vapnik, V. N. (1992). A training algorithm for optimal margin classifiers. COLT, 144–152. — the kernel trick
Platt, J. (1999). Probabilistic outputs for support vector machines. Advances in Large Margin Classifiers. — the calibration step the SVM requires
Niculescu-Mizil, A. & Caruana, R. (2005). Predicting good probabilities with supervised learning. ICML, 625–632. — which model families are calibrated, and in which direction
Ng, A. Y. & Jordan, M. I. (2001). On discriminative vs. generative classifiers. NeurIPS 14.