ML Arc, Foundations — SVM and the Discriminative Spectrum (R companion)¶
e1071::svm — max-margin, the kernel trick, and calibrating a classifier that has no probabilities¶
e1071::svm (a LIBSVM wrapper) is R's standard support vector machine. This companion reproduces the Python notebook: the SVM finds a maximum-margin boundary defined by a few support vectors, the RBF kernel draws nonlinear boundaries without modeling the features, and — because the SVM commits to no probability model — its scores must be Platt-calibrated to become probabilities. On the credit data, the generative (naiveBayes), probabilistic-discriminative (glm), and geometric-discriminative (svm) classifiers rank similarly but differ in exactly what they hand you.
1. Maximum margin and support vectors¶
On a 2-D two-class problem, svm(kernel="linear") finds the widest-margin separating line, fixed by a handful of support vectors. Points far from the boundary are irrelevant — the opposite of a generative classifier, which uses the whole class distribution.
suppressMessages(library(e1071))
set.seed(1); X<-rbind(matrix(rnorm(120,-1.2,0.9),60,2), matrix(rnorm(120,1.5,0.9),60,2)); y<-factor(rep(0:1,each=60))
df<-data.frame(x1=X[,1],x2=X[,2],y=y)
m<-svm(y~., data=df, kernel="linear", cost=1, scale=FALSE)
cat(sprintf("Max-margin SVM: %d support vectors of %d points define the boundary.\n", nrow(m$SV), nrow(df)))
options(repr.plot.width=7.5, repr.plot.height=5)
gx<-seq(min(X[,1])-1,max(X[,1])+1,length=200); gy<-seq(min(X[,2])-1,max(X[,2])+1,length=200)
grid<-expand.grid(x1=gx,x2=gy); dv<-attr(predict(m,grid,decision.values=TRUE),"decision.values")[,1]
plot(X[,1],X[,2],col=ifelse(y==0,"#2b6cb0","#c53030"),pch=19,xlab="x1",ylab="x2",main="SVM max-margin boundary + support vectors")
contour(gx,gy,matrix(dv,200,200),levels=c(-1,0,1),lty=c(2,1,2),lwd=c(1,2,1),add=TRUE,drawlabels=FALSE)
points(m$SV,cex=2.2,col="#2f855a",lwd=2)
legend("topleft",c("class 0","class 1","support vectors"),col=c("#2b6cb0","#c53030","#2f855a"),pch=c(19,19,1),bty="n")
Max-margin SVM: 8 support vectors of 120 points define the boundary.
2. The kernel trick — nonlinear boundaries¶
kernel="radial" (RBF) maps the features implicitly into a high-dimensional space and finds a linear margin there — a curved boundary in the original space — with no model of $p(x)$. On two concentric rings, which no line can separate, the RBF-SVM traces the correct circular boundary. (The kernel machinery is built from scratch in the SVM and Kernel Methods notebook.)
set.seed(0); n<-300; r<-c(runif(n/2,0,0.5),runif(n/2,0.8,1.2)); th<-runif(n,0,2*pi)
Xc<-cbind(r*cos(th)+rnorm(n,0,0.05), r*sin(th)+rnorm(n,0,0.05)); yc<-factor(rep(0:1,each=n/2))
dc<-data.frame(x1=Xc[,1],x2=Xc[,2],y=yc)
options(repr.plot.width=13, repr.plot.height=4.8); par(mfrow=c(1,2))
for(k in c("linear","radial")){
mm<-svm(y~., dc, kernel=k, cost=2)
gx<-seq(-1.4,1.4,length=200); grid<-expand.grid(x1=gx,x2=gx)
dv<-attr(predict(mm,grid,decision.values=TRUE),"decision.values")[,1]
plot(Xc[,1],Xc[,2],col=ifelse(yc==0,"#2b6cb0","#c53030"),pch=19,cex=.6,xlab="",ylab="",
main=sprintf("%s SVM (train acc %.2f)",k,mean(predict(mm,dc)==yc)))
contour(gx,gx,matrix(dv,200,200),levels=0,add=TRUE,lwd=2,drawlabels=FALSE) }
par(mfrow=c(1,1))
3. No native probabilities — and the credit comparison¶
svm returns a signed decision value, not a probability; probability=TRUE fits an internal Platt logistic to produce calibrated probabilities. We show the raw decision values are unbounded, then compare the three paradigms on the credit data — generative (naiveBayes), probabilistic-discriminative (glm), geometric-discriminative (svm). Their AUCs are close; the differences are in probabilities (Naive Bayes: wrong; SVM: none without Platt; logistic: native).
cd<-read.csv("credit_default.csv"); cd$default<-factor(cd$default)
set.seed(0); i<-sample(nrow(cd),0.7*nrow(cd)); tr<-cd[i,]; te<-cd[-i,]
trs<-tr[sample(nrow(tr),4000),] # SVM on a subsample
auc<-function(p,yv){ yv<-as.integer(yv)-1; rk<-rank(p); (sum(rk[yv==1])-sum(yv==1)*(sum(yv==1)+1)/2)/(sum(yv==1)*sum(yv==0)) }
sv<-svm(default~., trs, kernel="radial", probability=TRUE)
dv<-attr(predict(sv,te,decision.values=TRUE),"decision.values")[,1]
pp<-attr(predict(sv,te,probability=TRUE),"probabilities")[,"1"]
cat(sprintf("SVM raw decision values range: [%.2f, %.2f] -- NOT probabilities; Platt maps them to [0,1].\n", min(dv),max(dv)))
# Fit every paradigm on the SAME subsample the SVM had to use, so the comparison is of methods and not sample sizes.
nbs<-naiveBayes(default~.,trs); pnbs<-predict(nbs,te,type="raw")[,2]
lgs<-suppressWarnings(glm(default~.,trs,family=binomial)); plgs<-predict(lgs,te,type="response")
nb<-naiveBayes(default~.,tr); pnb<-predict(nb,te,type="raw")[,2]
lg<-suppressWarnings(glm(default~.,tr,family=binomial)); plg<-predict(lg,te,type="response")
res<-c(`Naive Bayes (generative)`=auc(pnbs,te$default), `Logistic (prob. discrim.)`=auc(plgs,te$default), `RBF SVM (geometric)`=auc(pp,te$default))
cat(sprintf("\nAUC by paradigm, all three on the same %d training rows:\n", nrow(trs))); print(round(res,3))
cat(sprintf("\nFor comparison, the two cheap models on all %d rows: Naive Bayes %.3f, logistic %.3f.\n",
nrow(tr), auc(pnb,te$default), auc(plg,te$default)))
cat("An RBF kernel is O(n^2), so fitting the SVM on a subsample while the linear models see everything is the usual\n")
cat("shortcut -- and it turns a comparison of paradigms into a comparison of sample sizes. Equalised, the ranking is\n")
cat("unchanged, so the shortcut was not driving it here. Worth checking rather than assuming.\n")
ece<-function(p,yv){ yv<-as.integer(yv)-1; b<-cut(p,seq(0,1,0.1),include.lowest=TRUE); e<-0
for(l in levels(b)){ m<-b==l; if(sum(m)>0) e<-e+mean(m)*abs(mean(yv[m])-mean(p[m])) }; e }
cat(sprintf("\nCalibration on the same footing: Naive Bayes ECE %.3f, logistic %.3f, Platt-scaled SVM %.3f.\n",
ece(pnbs,te$default), ece(plgs,te$default), ece(pp,te$default)))
cat("The SVM had no probabilities at all until Platt supplied them, and they come out BETTER calibrated than the ones\n")
cat("logistic produces natively. 'Native' is not the same as 'good' -- the Calibration notebook reaches the same\n")
cat("conclusion from the other side, finding logistic second-worst of four models on this dataset.\n")
cat(sprintf("\nAnd the sparsity from section 1 does not survive real data: %d support vectors of %d (%.0f%%), against 8 of 120 (7%%)\n",
nrow(sv$SV), nrow(trs), 100*nrow(sv$SV)/nrow(trs)))
cat("on the separable toy problem. When classes overlap, nearly every point is a boundary point.\n")
options(repr.plot.width=9, repr.plot.height=3.8)
# horiz=TRUE with las=1 writes the category names into the LEFT margin, and the default 4.1 lines
# clips names this long down to their last few characters -- reserve room, then restore.
op<-par(mar=c(4.2, 12.5, 3.0, 1.0))
# xpd=FALSE: barplot draws with xpd=TRUE by default, so with xlim starting at 0.6 the bars would
# run from 0 out across the left margin and sit underneath the category labels.
bp<-barplot(res, horiz=TRUE, las=1, col=c("#2f855a","#2b6cb0","#c53030"), xlim=c(0.6,0.76), xlab="test AUC", xpd=FALSE,
main="Credit: three paradigms, similar ranking")
text(res-0.01, bp, sprintf("%.3f",res), col="white", font=2)
par(op)
SVM raw decision values range: [-2.02, 1.68] -- NOT probabilities; Platt maps them to [0,1].
AUC by paradigm, all three on the same 4000 training rows:
Naive Bayes (generative) Logistic (prob. discrim.) RBF SVM (geometric)
0.719 0.710 0.712
For comparison, the two cheap models on all 21000 rows: Naive Bayes 0.735, logistic 0.712.
An RBF kernel is O(n^2), so fitting the SVM on a subsample while the linear models see everything is the usual
shortcut -- and it turns a comparison of paradigms into a comparison of sample sizes. Equalised, the ranking is
unchanged, so the shortcut was not driving it here. Worth checking rather than assuming.
Calibration on the same footing: Naive Bayes ECE 0.444, logistic 0.050, Platt-scaled SVM 0.017.
The SVM had no probabilities at all until Platt supplied them, and they come out BETTER calibrated than the ones
logistic produces natively. 'Native' is not the same as 'good' -- the Calibration notebook reaches the same
conclusion from the other side, finding logistic second-worst of four models on this dataset.
And the sparsity from section 1 does not survive real data: 1914 support vectors of 4000 (48%), against 8 of 120 (7%)
on the separable toy problem. When classes overlap, nearly every point is a boundary point.
4. Summary¶
e1071::svm reproduced the discriminative extreme: a maximum-margin boundary fixed by support vectors, the RBF kernel for nonlinear separation, and — since the SVM models no distribution — decision values that must be Platt-calibrated into probabilities. Alongside naiveBayes (generative) and glm (probabilistic-discriminative), the three complete the ladder, differing in what they output — Naive Bayes' probabilities are wrong, the SVM's have to be created by calibration, logistic's arrive without one.
Two corrections to the usual telling came out of running it. The comparison was equalised first: an $O(n^2)$ kernel makes it routine to fit the SVM on a subsample while the linear models see everything, which compares sample sizes rather than paradigms. Equalised, the ranking held. And logistic's native probabilities are not the best ones — the Platt-scaled SVM is better calibrated, and the Calibration notebook independently found logistic second-worst of four on this data. "Native" is not "good".
This closes the ML arc's foundations section: classifiers arranged by how much of the data-generating process they commit to, from generative (everything) through probabilistic-discriminative (the conditional) to geometric-discriminative (only the boundary). Cross-links: the kernel machinery and the kernel-ridge = Gaussian-process identity are in SVM and Kernel Methods; the Platt/calibration thread continues into Model Evaluation – Calibration. Guidance: SVM (RBF) for clean-margin, high-dimensional problems where robustness matters and you can calibrate afterward; logistic for probabilities out of the box; generative when data are scarce.