I5 · Machine Learning Foundations

Support Vector Machines

A support vector machine (SVM) is a classifier that finds the boundary with the maximum margin, the largest distance to the nearest training points of each class. Those nearest points are the support vectors, and only they determine the boundary. A soft-margin SVM allows some points to violate the margin, controlled by the parameter C. Using the kernel trick, an SVM can compute similarities as if the data were mapped into a higher-dimensional space, producing curved boundaries; the radial basis function (RBF) kernel with parameter gamma is the most common choice.

12 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

Many different lines can separate cats from dogs in a data set. Which one should you trust on new animals? A support vector machine picks the line that leaves the widest gap on both sides, and with a clever trick it can draw curved boundaries too.

A support vector machine (SVM) is a classifier that finds the boundary with the maximum margin, the largest distance to the nearest training points of each class. Those nearest points are the support vectors, and only they determine the boundary. A soft-margin SVM allows some points to violate the margin, controlled by the parameter C. Using the kernel trick, an SVM can compute similarities as if the data were mapped into a higher-dimensional space, producing curved boundaries; the radial basis function (RBF) kernel with parameter gamma is the most common choice.

In simple words

Imagine building a road between two villages. You want the widest road possible that does not touch any house. The houses nearest the road edges decide where the road goes; houses far away do not matter. That road is an SVM boundary.

Overview

  • A support vector machine (SVM) finds the boundary with the widest margin between classes.
  • Only the support vectors, the points nearest the boundary, determine it.
  • C controls how strictly training points must be on the right side.
  • Kernels let SVMs draw curved boundaries by comparing points through similarity functions.
  • Features must be scaled, and kernel SVMs become slow beyond about 100,000 rows.
  • SVMs were developed by Vladimir Vapnik and colleagues, with the soft-margin form published in 1995.

Why it matters

SVMs give strong accuracy on small and medium data sets, especially with many features such as text, and the margin idea explains why some boundaries generalise better than others. They were among the most successful methods before deep learning and remain a useful tool and baseline.

Key terms

Hyperplane
A flat boundary: a line in 2D, a plane in 3D, and its generalisation in more dimensions.
Margin
The gap between the boundary and the closest points of each class.
Support vectors
The training points on or inside the margin that define the boundary.
C parameter
Penalty for margin violations: large C fits training data tightly, small C allows a wider, softer margin.
Kernel
A function measuring similarity between points, such as linear, polynomial or RBF.
Gamma
For the RBF kernel, how far each training point's influence reaches; large gamma means very local, wiggly boundaries.

Types

SVMs come in several types by kernel and by task. These are the main ones.

1. Linear SVM

An SVM with a straight-line (hyperplane) boundary.

  • How it works: LinearSVC optimises the hinge loss directly and scales to millions of examples and features.
  • Example: Classifying news articles with 50,000 TF-IDF features.
  • When to use it: Use it for text and other high-dimensional data, where a linear boundary is usually enough. Tune C with cross-validation for the best margin.

2. RBF (Gaussian) kernel SVM

An SVM using the radial basis function kernel K(a, b) = exp(−γ‖a − b‖²).

  • How it works: Each training point influences a region whose size is set by γ; large γ makes very wiggly boundaries.
  • Example: Separating points inside and outside a circle: about 93% test accuracy versus 47% for a linear SVM.
  • When to use it: Use it as the default kernel for small and medium non-linear problems, tuning C and γ.

3. Polynomial and other kernels

Kernels such as polynomial (a · b + c)ᵈ and sigmoid.

  • How it works: A polynomial kernel of degree d behaves like using all feature products up to degree d without computing them.
  • Example: Image recognition with degree-2 or degree-3 polynomial kernels in the 1990s.
  • When to use it: Use them when you know feature interactions of a certain order matter; RBF is usually tried first.

4. Support vector regression (SVR)

An SVM for predicting numbers.

  • How it works: It fits a tube of width ε around the predictions, ignoring errors inside the tube and penalising those outside.
  • Example: Predicting electricity demand while ignoring small fluctuations.
  • When to use it: Use it for regression on small to medium data where robustness to small errors helps.

5. One-class SVM

An SVM trained only on normal data to detect unusual points.

  • How it works: It learns a boundary around the normal data; points outside are flagged as anomalies.
  • Example: Detecting faulty machine readings when only healthy readings are available for training.
  • When to use it: Use it for novelty detection on moderate-sized data with scaled features.

Explained in detail

Maximum margin

For data that two classes can be separated by a straight line, there are infinitely many separating lines. The SVM chooses the one that maximises the margin. Intuitively, a boundary that passes close to training points is fragile: a new point slightly different from a training example could fall on the wrong side. A wide margin leaves room for such variation. Mathematically, the boundary is w·x + b = 0, the margin edges are w·x + b = ±1, and the margin width is 2 ÷ ‖w‖, so maximising the margin means minimising ‖w‖ subject to every point being on the correct side. Only the support vectors appear in the solution, which makes the model compact.

Soft margins and the C parameter

Real data overlaps and contains mislabelled points, so a perfect separation may be impossible or may overfit. The soft-margin SVM lets points fall inside the margin or on the wrong side, paying a penalty measured by the hinge loss. C balances margin width against these penalties. With C = 100, the SVM tries hard to classify every training point correctly, producing a narrow margin that may follow noise. With C = 0.01, it accepts more mistakes for a wide, stable margin. C is tuned by cross-validation, usually over values like 0.01, 0.1, 1, 10 and 100.

The kernel trick

Some data cannot be separated by any straight line: points of one class inside a circle and the other outside it. If we add a third feature z = x² + y², the inner points sit low and the outer points sit high, and a flat plane separates them. The kernel trick lets the SVM use such transformations without computing them, because its training depends only on similarities between pairs of points. The RBF kernel, K(a, b) = exp(−γ‖a − b‖²), gives high similarity to nearby points and near zero to distant ones, and can model very flexible boundaries. In the code below, the linear kernel scores only about 47 per cent on test circles, no better than guessing, while the RBF kernel reaches 93 per cent; pushing C to 100 and gamma to 50 gives 100 per cent on training data but only 89 per cent on test data, a clear sign of overfitting.

SettingEffect
Linear kernelStraight boundary; fast; great for text with thousands of features
RBF kernel, small gammaSmooth, gently curved boundary
RBF kernel, large gammaWiggly boundary around individual points; risk of overfitting

How it works

  1. Standardise the features; SVMs are very sensitive to scale.
  2. Start with a linear kernel for text or many features, RBF otherwise.
  3. Tune C (and gamma for RBF) with cross-validation on a logarithmic grid.
  4. Inspect the number of support vectors: a huge share may signal overlap or poor settings.
  5. Calibrate if you need probabilities, since SVMs output scores.

The maths

Decision function
Margin width
RBF kernel
Controls how quickly similarity falls with distance

Take a one-feature example: class A has points at x = 1 and 2, class B at x = 4 and 5. Any threshold between 2 and 4 separates them, but the SVM picks the middle, x = 3, because it is 1 unit from the nearest point on each side, the largest possible margin. The support vectors are x = 2 and x = 4; moving the point at x = 1 to x = 0 changes nothing. Written as w·x + b with margin edges at ±1, we need 2w + b = −1 and 4w + b = 1, giving w = 1 and b = −3, and margin width 2 ÷ |w| = 2.

In code

svm_kernels.py
import numpy as np
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

# The worked example: support vectors and the boundary at x = 3
X1 = np.array([[1], [2], [4], [5]]); y1 = np.array([0, 0, 1, 1])
svm1 = SVC(kernel="linear", C=1000).fit(X1, y1)
print("Support vectors:", svm1.support_vectors_.ravel(), " boundary at x =", round(-svm1.intercept_[0] / svm1.coef_[0][0], 2))

# Circles: a straight line fails, the RBF kernel succeeds
X, y = make_circles(n_samples=400, noise=0.2, factor=0.5, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
for kernel, C, gamma in [("linear", 1, "scale"), ("rbf", 1, "scale"), ("rbf", 100, 50)]:
    model = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=C, gamma=gamma)).fit(X_tr, y_tr)
    svc = model[-1]
    print(f"{kernel:<6} C={C:<4} gamma={str(gamma):<6} train={model.score(X_tr, y_tr):.2f} test={model.score(X_te, y_te):.2f} support vectors={svc.n_support_.sum()}")

Output

Support vectors: [2. 4.]  boundary at x = 3.0
linear C=1    gamma=scale  train=0.51 test=0.47 support vectors=297
rbf    C=1    gamma=scale  train=0.90 test=0.93 support vectors=112
rbf    C=100  gamma=50     train=1.00 test=0.89 support vectors=252

Try it yourself

Sorting good and faulty bolts

A factory measures length and head width of bolts and wants to reject faulty ones.

Match the SVM idea

What does each term mean?

  • Support vectors
  • Large C
  • RBF kernel
  • Large gamma

Common mistakes

  • Forgetting to scale features before an SVM.
  • Using a large C and large gamma without validation, causing overfitting.
  • Running a kernel SVM on millions of rows.

Check yourself

Question 1

1. Which training points determine an SVM's boundary?

Question 2

2. What does the kernel trick allow?

Question 3

3. An RBF SVM scores 100% on training and 70% on test. A sensible change is…

Summary

  • An SVM picks the maximum-margin boundary, defined only by the support vectors.
  • C trades margin width against training errors; kernels such as RBF allow curved boundaries.
  • Scale features, tune C and gamma by cross-validation, and prefer linear SVMs for very large or text data.

Finished reading?

Sources: James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning (free book) · scikit-learn user guide