Support Vector Machines

01 Learning Outcomes

02 Why This Unit Matters

Units 7 and 8 built models that partition feature space with axis-aligned cuts. Support vector machines take a fundamentally different approach: they look for the single boundary that separates the classes with the largest possible safety margin. That geometric objective turns out to be far more robust than it first appears, and it generalises to boundaries that are not straight lines at all.

SVMs also introduce two ideas that recur throughout machine learning. The first is that only a few training points matter — the ones nearest the boundary. The second is the kernel trick, a mathematical shortcut that lets a linear algorithm behave non-linearly. Both reappear in later units and in modern deep learning.

03 Reading Roadmap

Work through the eight tutorials in order. Tutorials t-01 and t-02 establish the geometry; t-03 relaxes it to handle real, non-separable data; and t-04 introduces the kernel trick, which is the conceptual highlight of the unit. Take the hand calculations in t-02 slowly — they make everything afterwards concrete.

04 Topics & Key Concepts

Terms you must be able to define

Hyperplane

A flat decision boundary of one dimension fewer than the feature space. In 2-D it is a line; in 3-D it is a plane. Written as w·x + b = 0.

Weight vector (w)

The vector normal (perpendicular) to the hyperplane. Its direction sets the orientation of the boundary; its length controls the margin.

Margin

The distance between the decision boundary and the nearest training points on either side. The SVM's objective is to maximise it.

Support vectors

The training points that lie exactly on the margin — the ones that satisfy |w·x + b| = 1. Only these points determine the boundary; removing any other point leaves it unchanged.

Hard margin

A formulation that requires every training point to be classified correctly and lie outside the margin. Only possible when the data are linearly separable.

Soft margin

A formulation that permits margin violations at a cost. Used for almost all real datasets.

Slack variable (ξ)

A per-example allowance for how far a point may intrude into the margin or cross the boundary. Zero for well-classified points outside the margin.

C parameter

The penalty for total slack. Large C punishes violations harshly (narrow margin, overfitting risk); small C tolerates them (wide margin, underfitting risk).

Hinge loss

The loss function max(0, 1 − y(w·x + b)), which is zero for correctly classified points outside the margin and grows linearly for violations.

Kernel trick

Rewriting the SVM optimisation so it depends only on dot products between examples, then substituting a kernel function K(xᵢ, xⱼ) that implicitly computes a dot product in a higher-dimensional space.

RBF (Gaussian) kernel

K(xᵢ, xⱼ) = exp(−γ‖xᵢ − xⱼ‖²). Maps to an infinite-dimensional space and can produce arbitrarily curved boundaries. Controlled by gamma.

Gamma (γ)

The RBF kernel's reach parameter. Large gamma means each point influences only its immediate neighbourhood (wiggly boundary); small gamma means influence reaches further (smooth boundary).

05 Worked Example: Finding the Maximum Margin by Hand

Consider a small, perfectly separable 2-D dataset. The positive class contains two points and the negative class contains two points, arranged symmetrically along the line y = x.

06 Learning Activities

Notebook

The core of the notebook is a scaled pipeline with a grid search:

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV

# Scaling is not optional for SVMs — it is part of the model.
pipe = make_pipeline(
    StandardScaler(),
    SVC(kernel="rbf", probability=True, random_state=42),
)

param_grid = {
    "svc__C":     [0.1, 1, 10, 100],
    "svc__gamma": ["scale", 0.01, 0.1, 1],
}

search = GridSearchCV(pipe, param_grid, cv=5, scoring="f1", n_jobs=-1)
search.fit(X_train, y_train)

print("Best params:", search.best_params_)
print("Best CV F1: ", round(search.best_score_, 3))

best = search.best_estimator_
print("Support vectors per class:", best[-1].n_support_)

07 Self-Check

Answer these before moving on. If any feel uncertain, revisit the linked tutorial.

08 Assessment Activities

• Quiz 9 Support Vector Machines — 12 questions, auto-graded Due end of week 9
• Module 3 Review Tree vs forest vs SVM — comparative report Due week 10
• Capstone Justify your choice of algorithm family for your capstone problem End of course

09 Summary