Supervised Learning Model Evaluation

01 Why this unit

Everything you have built so far — a fitted regression, a trained classifier — is only as good as the evidence you bring to judge it. This unit teaches you how to gather that evidence properly, and how to tell when a model that looks impressive is quietly failing.

The central lesson is uncomfortable: a single number is almost never a sufficient evaluation. Accuracy hides class imbalance. Precision hides missed positives. ROC-AUC hides poor performance in the operating region you actually care about. A test score computed on data the model has already seen hides everything. Evaluation is therefore not a final step you perform once the modelling is done — it is a discipline that shapes how you split your data, how you tune your model, and how you decide when to stop.

02 Learning outcomes

By the end of Unit 6 you should be able to do everything listed below without referring back to the notes. These outcomes are the ones the quiz and Assignment 3 are written against, and they carry forward into the capstone.

03 Reading roadmap

Work through the tutorials in order. Tutorials 1–5 build the metric vocabulary; Tutorials 6–7 build the validation discipline; Tutorial 8 puts both together in a full model-selection workflow.

Suggested study sequence

1. Read the concept, don't skim it Sections 1–5 of each tutorial carry the idea. Slow down on the worked example.
2. Run the code yourself Every notebook is short. Type it out at least once before you copy-paste it.
3. Do the self-check before the answer reveal Section 7 exists to catch the misunderstanding while it is still cheap to fix.
4. Attempt the practice exercises and problem set Sections 8 and 9 are where the skill actually forms.
5. Write two sentences in the discussion Section 10 asks you to defend an evaluation choice, not just compute a metric.

04 Tutorials

Eight self-contained pages. Each follows the same twelve-section structure — objectives, motivating example, concept, visual, worked example, interactive activity, self-check, practice exercises, problem set, discussion, summary and further reading.

05 Learning activities

Reading alone will not get you to the outcomes. These are the activities that will.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(
    n_samples=2000, n_features=12, weights=[0.98, 0.02], random_state=42
)

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for metric in ("accuracy", "precision", "recall", "f1", "roc_auc"):
    scores = cross_val_score(pipe, X, y, cv=cv, scoring=metric)
    print(f"{metric:>9}: {scores.mean():.3f}")

 accuracy: 0.981
precision: 0.556
   recall: 0.425
      f1: 0.481
  roc_auc: 0.928

Accuracy of 98.1% looks like a finished project. Recall of 42.5% means the model misses more than half of the positive cases. Both numbers describe the same classifier — which one you report determines whether anyone notices.

06 Assessment

Unit 6 completes Module 2. It contributes through the final quiz of the module and through the evaluation sections of Assignment 3, which are assessed here.

What the quiz will ask you to do

Assignment 3 — evaluation and reporting sections

• Part A Problem framing and data preparation — target definition, class balance, leakage check Unit 5
• Part B Model fitting and evaluation — confusion matrix, metric selection, cross-validation, curves Unit 6
• Part C Threshold justification, error analysis and honest reporting — costs, uncertainty, limitations Unit 6

Module 2 checkpoint

Unit 6 is the last unit of Module 2. Before moving to Module 3, check that you can complete each of the following. They are the Module 2 outcomes, and the capstone assumes all of them.

07 Summary

Choosing a metric

Key terms

Confusion matrix

A four-cell table counting true positives, false positives, true negatives and false negatives.

Precision

TP / (TP + FP) — the proportion of positive predictions that were correct. Also called positive predictive value.

Recall

TP / (TP + FN) — the proportion of actual positives that were found. Also called sensitivity or true positive rate.

Specificity

TN / (TN + FP) — the proportion of actual negatives that were correctly identified. The complement of the false positive rate.

F1-score

The harmonic mean of precision and recall, which penalises a model that is strong on one and weak on the other.

ROC curve

A plot of true positive rate against false positive rate as the decision threshold varies from 1 to 0.

AUC

The area under the ROC curve. Equivalently, the probability that a randomly chosen positive is ranked above a randomly chosen negative.

Precision–recall curve

A plot of precision against recall across thresholds. Preferred over ROC when the positive class is rare.

Cross-validation

Partitioning the data into k folds, training on k−1 and validating on the held-out fold, then averaging. Gives a more stable estimate than a single split.

Data leakage

Any situation where information from outside the training fold influences the model or its evaluation, producing scores that will not generalise.

Bias–variance tradeoff

The tension between a model too simple to capture the signal (high bias, underfitting) and one so flexible it fits the noise (high variance, overfitting).

Learning curve

A plot of training and validation performance against training set size, used to diagnose whether more data would help.