Support Vector Machine

Definition

A support vector machine (SVM; Boser-Guyon-Vapnik 1992) is a classifier that separates two classes with the hyperplane maximizing the margin — the distance from the boundary to the nearest training points. The solution depends only on those nearest points, the support vectors, and — via the kernel trick — the same construction yields nonlinear boundaries by implicitly mapping the data into a high-dimensional feature space. It is a convex optimization problem with a single global optimum and a generalization theory grounded in the VC dimension.

Key Ideas

• Maximum margin. Among all separating hyperplanes, choose the one farthest from the closest points; equivalently, minimize $\tfrac12\lVert w\rVert^2$ subject to $y_i(w'x_i+b)\ge 1$. A larger margin means lower capacity and, by structural risk minimization, better expected generalization.
• Support vectors. Only the training points on the margin have nonzero dual weights; the decision function $f(x)=\sum_i \alpha_i y_i K(x_i,x)+b$ is a sparse linear combination of these support vectors, so most of the data can be discarded after training.
• Kernel trick. Because the problem depends on the data only through inner products, replacing $x_i'x_j$ with a kernel $K(x_i,x_j)$ (polynomial, radial basis function, …) fits a maximum-margin boundary in a high-dimensional feature space without ever computing the map — turning a linear method into a flexible nonlinear one.
• Capacity control via VC dimension. Generalization is bounded by the VC dimension of the margin classifier and by the leave-one-out estimator (expected error $\le$ expected fraction of support vectors), decoupling generalization from the raw parameter count.
• Soft margin (Cortes-Vapnik 1995). The original hard-margin formulation required error-free separation; the "support-vector network" paper extends it with slack variables $\xi_i\ge0$ to tolerate non-separable data, minimizing $\tfrac12\lVert w\rVert^2 + C\sum_i\xi_i$ — trading margin width against misclassification through the penalty $C$. This is the formulation used in practice.

Why It Matters

• Principled nonlinear classification. SVMs pair a convex, globally-solvable optimization with an explicit generalization theory — the combination that made them the dominant off-the-shelf classifier through the 2000s.
• The kernel idea is portable. Kernelization spread far beyond classification (kernel regression, kernel PCA, Gaussian processes), making this paper a root of a whole methodological family.
• A benchmark for the ML cluster. It is the margin/kernel counterpart to tree ensembles (random forests) and neural networks (attention) in empirical machine learning, including asset-pricing applications.

Open Questions

• Scaling. The kernel (Gram) matrix is $O(n^2)$ in memory and training is worse, so plain SVMs struggle on very large $n$ — a key reason deep networks overtook them on big data.
• Kernel and hyperparameter choice. Performance hinges on the kernel family and its parameters (and $C$), selected by cross-validation with no fully automatic rule.
• Probabilities and multiclass. SVMs output scores, not calibrated probabilities; probability estimates (Platt scaling) and multiclass schemes (one-vs-rest/one-vs-one) are add-ons rather than native.

Related

• Classification and Regression Trees (CART)
• Random Forest
• Machine Learning in Asset Pricing
• Low-Rank Approximation
• Boser-Guyon-Vapnik (1992) A Training Algorithm for Optimal Margin Classifiers
• Cortes-Vapnik (1995) Support-Vector Networks
• Vladimir N. Vapnik
• Corinna Cortes
• Isabelle M. Guyon
• Bernhard E. Boser