Definition
Of all the lines that separate two groups, which is the safest?
A support vector machine (SVM) finds the hyperplane that separates two classes with the largest margin, the distance to the nearest training points, which are called support vectors. The soft-margin SVM allows some violations, controlled by C. The kernel trick replaces dot products with a kernel function, such as the RBF kernel, letting the SVM learn non-linear boundaries without computing the high-dimensional features explicitly.
In simple words
Draw the widest possible road between the two groups; the decision line runs down the middle. Only the points on the edge of the road matter. Large C insists on few points inside the road (risking overfitting); small C allows a wider, more forgiving road. Kernels let the road bend.
Why it matters
SVMs give strong results on small and medium datasets with many features, such as text and genomics, and their maths, margins, duality and kernels, underpins much of modern learning theory.
Key terms
- Margin
- The distance between the decision boundary and the nearest training points of each class.
- Support vectors
- The training points on or inside the margin that alone determine the boundary.
- Soft margin and C
- Allowing some points to violate the margin; C controls the penalty for violations.
- Kernel trick
- Computing similarities as if data were mapped to a higher-dimensional space, without doing the mapping.
- RBF kernel and gamma
- K(a, b) = exp(−γ‖a − b‖²); gamma controls how far each point's influence reaches.
- Hinge loss
- max(0, 1 − y·f(x)): zero for points correctly beyond the margin.
Explained in detail
The widest street
Many lines can separate two classes; a support vector machine chooses the one with the widest margin, like the widest possible street between two groups of houses. Only the points touching the street edges, the support vectors, matter; moving any other point, as long as it stays outside the street, does not change the boundary. A wide margin tends to generalise well. Real data overlaps, so the soft-margin SVM allows some points inside or across the street, paying a penalty weighted by C. Large C insists on classifying training points correctly, giving a narrow margin and risking overfitting; small C tolerates mistakes for a wider, smoother boundary.
Kernels for curved boundaries
When classes cannot be separated by a straight line, such as points inside and outside a circle, mapping the data to more dimensions can make them separable: adding x² + y² as a third feature lifts the inner points below the outer ones, and a flat plane then separates them. The kernel trick lets the SVM work with such mappings implicitly, because training needs only dot products between points, which a kernel function provides directly. The RBF (Gaussian) kernel corresponds to an infinite-dimensional mapping and can model very flexible boundaries. Its gamma parameter sets the reach of each training point: large gamma gives a wiggly boundary that can overfit; small gamma gives a smooth one.
| Kernel | Use when |
|---|---|
| Linear | Many features relative to rows, such as text with TF-IDF |
| RBF | A good default for moderate-sized tabular data with curved boundaries |
| Polynomial | Known polynomial interactions between features |
SVMs were developed largely by Vladimir Vapnik and colleagues in the 1990s and were the leading method for many tasks, such as handwritten-digit and text classification, before deep learning. They need scaled features, and C and gamma must be tuned together by grid search on a logarithmic scale. Kernel SVMs scale roughly between the square and cube of the number of rows, so they become slow beyond about 100,000 rows; LinearSVC and SGD-based linear SVMs handle millions. SVMs output scores rather than probabilities; use probability=True or a calibration step if probabilities are needed. Support vector regression applies the same margin idea to predict numbers.
How it works
| Kernel | Shape of boundary | Key parameter |
|---|---|---|
| linear | Straight hyperplane | C |
| poly | Polynomial curves | C, degree |
| rbf | Smooth, flexible regions | C, gamma (larger = wigglier) |
The maths
In code
Linear and RBF SVMs on two interleaving half-moons, and the effect of C on the number of support vectors.
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_moons(n_samples=400, noise=0.25, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
for kernel in ["linear", "rbf"]:
model = make_pipeline(StandardScaler(), SVC(kernel=kernel)).fit(X_tr, y_tr)
print(f"{kernel:>6} kernel: test accuracy {model.score(X_te, y_te):.3f}")
print(" C support vectors test accuracy")
for C in [0.01, 1, 100]:
svc = SVC(kernel="rbf", C=C).fit(StandardScaler().fit(X_tr).transform(X_tr), y_tr)
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=C)).fit(X_tr, y_tr)
print(f"{C:>5} {svc.n_support_.sum():>15} {model.score(X_te, y_te):13.3f}")Output
linear kernel: test accuracy 0.908
rbf kernel: test accuracy 0.933
C support vectors test accuracy
0.01 278 0.750
1 88 0.933
100 55 0.942Try it yourself
Which kernel fits?
Match each dataset description to the kernel you'd try first.
- Text with 50,000 word features
- Two moons curling round each other
- A known quadratic relationship
Common mistakes
- Running an RBF SVM on unscaled data.
- Tuning C without gamma, or on a linear grid.
- Using SVMs on millions of rows, where training time grows faster than linearly.
Check yourself
Summary
- SVMs maximise the margin 2/‖w‖ between classes.
- C trades margin width against violations.
- Kernels such as RBF give non-linear boundaries; scale and tune C with gamma.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: scikit-learn: Support vector machines · Hastie et al., The Elements of Statistical Learning