Unit 13 — Anomaly Detection | COMP328: Fundamentals of Machine Learning

01 Learning outcomes

By the end of Unit 13 you should be able to do the following. Each outcome is assessed somewhere — the mapping is given in Section 05.

Knowledge — explain

Define an anomaly and distinguish it from noise and from a data-entry error.
Classify anomalies as point, contextual, or collective, and give an example of each.
Distinguish outlier detection from novelty detection.
Explain why anomaly detection is an extreme class-imbalance problem and why accuracy is meaningless as a metric.
Describe how statistical, distance-based, density-based, and isolation-based methods differ in their notion of "unusual".

Skills — apply

Detect outliers with z-scores and the IQR rule, and explain when each is appropriate.
Implement Local Outlier Factor and Isolation Forest with scikit-learn.
Tune the contamination parameter and neighbourhood size, and explain the trade-off each one controls.
Evaluate a detector using precision, recall, F1, and PR-AUC rather than accuracy.
Build a simple streaming monitor that scores new observations against a fitted detector.
Report a detector's performance honestly when labels are incomplete.

Professional competencies — evaluate

Recognise that an anomaly flag is a decision with consequences — every false positive costs someone time or money.
Assess the human impact of automated fraud and risk scoring, including who bears the cost of errors.
Identify and mitigate alert fatigue in a monitoring system.
Document thresholds, contamination settings, and detector versions so that alerts can be audited.

02 Why anomaly detection matters

Almost every operational system you can name has a failure mode that is rare by design. Card fraud runs at a fraction of a percent of transactions. A manufacturing defect appears in a handful of units per thousand. An intrusion may happen once in a year of logs. The entire point of a security or reliability system is to catch these events — and the entire difficulty is that there is almost nothing to learn from.

This is what separates anomaly detection from the rest of the course. In supervised learning you have a label for every example, balanced or not. Here, the anomalous class is often so small that a supervised model cannot be trained at all. You are forced to define what normal looks like and treat everything sufficiently far from it as suspicious. That definition is a modelling choice, and it has real consequences.

The cost asymmetry

In most anomaly detection problems, the two error types do not cost the same. Missing a fraudulent transaction can cost thousands. Flagging a legitimate one costs a blocked card, an angry customer, and a support ticket. Which error you weight more is a business decision, not a statistical one — and it must be made before you tune your threshold, not after. A detector tuned to maximise F1 with no regard for this asymmetry is optimising the wrong thing.

The three anomaly types on one series. The point anomaly is extreme on its own — no context needed. The collective anomaly is five consecutive values that are each only mildly unusual, but together form a sustained deviation. The contextual anomaly sits comfortably inside the global normal range and would be invisible to any detector that ignores its neighbours. Recognising which type you are hunting determines which algorithm you should reach for.

Common misconception

"Anomaly" is not a property of a data point — it is a property of a data point relative to a model of normality. Change the model and you change which points are anomalous. This is why two analysts can run two perfectly valid detectors on the same dataset and get two different alert lists, and why "the algorithm found it" is never a sufficient justification for acting on a flag. You must be able to say what "normal" meant.

03 Reading roadmap

The nine tutorials below are designed to be read in order. T-01 and T-02 define the problem and its variants; T-03 to T-06 cover the four main detector families; T-07 to T-09 deal with evaluation, implementation, and a full worked case study.

T-01

What an anomaly is

Normality as a modelling assumption, outliers versus noise, and why labels are usually missing.

T-02

Point, contextual, and collective anomalies

The three anomaly types, plus the distinction between outlier detection and novelty detection.

T-03

Statistical detectors

Z-scores, the IQR rule, Gaussian models, and the assumption of a distribution you rarely have.

T-04

Distance and density: kNN and LOF

Local density, reachability distance, and why a global threshold fails on varying-density data.

T-05

Isolation Forest

Isolating points with random splits, path length as an anomaly score, and why it scales so well.

T-06

One-Class SVM and autoencoders

Boundary-based detection, reconstruction error, and a bridge forward to Module 5.

T-07

Evaluating detectors

Precision, recall, PR-AUC, and why a 99.5%-accurate detector can be completely useless.

T-08

Detection in scikit-learn

A complete workflow: fitting, scoring, threshold selection, and monitoring new observations.

T-09

Case study: fraud and intrusion

Two end-to-end analyses — card transactions and network logs — with an explicit cost model.

Study tip

T-07 is the most important tutorial in this unit, and the one most students rush. Every detector in T-03 to T-06 produces a continuous score; the algorithm is the easy part. Choosing a threshold, and defending that choice in terms of the two error costs, is what makes an anomaly detection system useful. Work the precision–recall calculations in T-07 by hand before you write any code.

04 Learning activities

Each tutorial contains a worked example and a self-check. The activities below are the ones that require you to write and run code.

Activity 13.1 — Z-score versus IQR

Take the one-dimensional sensor series from T-03 and flag outliers twice: once using a z-score threshold of 3, and once using the IQR rule with a multiplier of 1.5. Compare the two flag sets. Then replace a single normal value with an extreme reading and repeat. Explain in three sentences why the z-score method's behaviour changed and the IQR method's did not.

Activity 13.2 — Four detectors, one dataset

Load the cardiotocography or breast_cancer dataset from scikit-learn and treat the minority class as "anomalous". Fit four detectors: Isolation Forest, Local Outlier Factor, One-Class SVM, and a simple z-score baseline. For each, set the contamination parameter to the true anomaly rate and report precision, recall, and PR-AUC on a held-out split. Rank the detectors and explain which assumptions each one made.

Activity 13.3 — Thresholds and cost

Using the best detector from Activity 13.2, plot the precision–recall curve. Then assume a false negative costs 100× a false positive, and compute the cost of operating at every threshold on the curve. Identify the cost-minimising threshold and compare it to the threshold that maximises F1. Write one paragraph explaining the difference to a non-technical stakeholder.

unit-13-anomaly-detection.ipynb Starter notebook containing sensor and transaction datasets, detector scaffolding, and plotting helpers for activities 13.1–13.3. Open workflow

05 Assessment activities

10%

Unit 13 quiz

12 questions covering anomaly types, statistical and density-based detectors, Isolation Forest, and imbalanced evaluation. Unlimited attempts; highest score counts.

15%

Assignment 4

Unsupervised learning. Your submission must include an anomaly detection component with a justified threshold and an explicit discussion of error costs.

—

Self-checks

Embedded in every tutorial. Ungraded, but they are the fastest way to find out whether you have actually understood the previous section.

Outcome-to-assessment mapping

Where each learning outcome is assessed
Outcome	Quiz	Assignment 4	Capstone
Define and classify anomalies	Q1–Q3	Report §1	Methodology
Apply statistical detectors	Q4–Q5	Code	Model building
Apply Isolation Forest and LOF	Q6–Q9	Code + results	Model building
Evaluate with imbalanced metrics	Q10–Q11	Report §3	Evaluation
Communicate alerts and costs responsibly	Q12	Report §4	Ethics analysis

Quiz 13 Anomaly detection — 12 questions, 30 minutes Revise

Assignment 4 Unsupervised Learning: clustering, dimensionality reduction, association rules, and anomaly detection Open brief

06 Key terms

Anomaly / outlier: An observation that deviates so far from the pattern of the remaining data that it is reasonable to suspect it was generated by a different mechanism.
Normality model: The explicit or implicit description of what "typical" looks like. Every anomaly detector has one, even if it is never stated.
Point anomaly: A single observation that is extreme with respect to the global distribution — the classic outlier.
Contextual anomaly: An observation that is unremarkable globally but unusual given its context — a time of day, a season, a location, or a neighbouring sequence.
Collective anomaly: A sequence of observations that are individually unremarkable but collectively form a deviating pattern.
Outlier detection: Finding anomalies in a dataset that is assumed to already contain them. The model is fitted on contaminated data.
Novelty detection: Fitting a model of normality on clean training data, then flagging departures from it in new observations.
Contamination: The proportion of anomalies a detector is told to expect. In scikit-learn it determines the decision threshold.
Local Outlier Factor (LOF): A density-based score comparing a point's local density to that of its neighbours. Values substantially above 1 indicate a local outlier.
Isolation Forest: An ensemble of random trees that isolates points by repeated random splitting. Anomalies require fewer splits to isolate, giving shorter path lengths.
One-Class SVM: A boundary-based method that learns a decision surface enclosing the normal data in a kernel-induced feature space.
Reconstruction error: For an autoencoder, the difference between an input and its reconstruction. High error indicates the model has not seen anything like it before.
PR-AUC: The area under the precision–recall curve. The standard summary metric for heavily imbalanced detection problems, where ROC-AUC is misleadingly optimistic.
Alert fatigue: The degradation of human response that occurs when a monitoring system produces more alerts than operators can meaningfully investigate.

07 Summary

What you should take away

An anomaly is defined relative to a model of normality. State your model, or you cannot defend your alerts.
There are three structural types — point, contextual, and collective — and they require different detectors. A method that finds point anomalies will miss a sustained collective deviation entirely.
Statistical methods such as z-scores and IQR are fast and interpretable, but assume a distribution and a single global notion of "far". The IQR rule is far more robust to extreme values than the z-score.
Density-based methods such as LOF handle clusters of varying density, which global thresholds cannot.
Isolation Forest is usually the best default for tabular data: fast, scalable, robust to irrelevant features, and requiring almost no assumptions about the data distribution.
Accuracy is worthless here. A detector that flags nothing achieves over 99% accuracy on a typical fraud dataset. Use precision, recall, and PR-AUC.
The threshold is a business decision, driven by the relative cost of the two error types — not a statistical one.
A detector that produces more alerts than a human can review has failed, regardless of its PR-AUC. Design for alert volume, not just detection quality.

End of Module 4

This is the final unit of Module 4. You have now covered the four pillars of unsupervised learning: clustering, dimensionality reduction, association learning, and anomaly detection. Once you have scored at least 80% on the unit quiz and completed Assignment 4, continue to Module 5 — Neural Networks and Deep Learning, beginning with Unit 14 — Neural Networks.

08 Frequently asked questions

How is this different from removing outliers in preprocessing?

The technique is often the same; the intent is opposite. In preprocessing (Unit 2), you detect outliers in order to delete them, so that they stop distorting a model fitted on the remaining data. Here, the outliers are the output. You are not cleaning the dataset — you are producing a ranked list of suspicious observations for a human to investigate. Same arithmetic, completely different purpose, and the evaluation criteria differ accordingly.

Why can't I just use accuracy to evaluate my detector?

Because of the base rate. Suppose 0.5% of transactions are fraudulent. A model that predicts "not fraud" for every single transaction is 99.5% accurate and catches precisely zero fraud. Accuracy is dominated by the majority class, so it rewards the trivial solution. Use precision (of the transactions you flagged, how many were real?), recall (of the real frauds, how many did you catch?), and PR-AUC to summarise across all thresholds. ROC-AUC is also misleading here — the false-positive rate stays small simply because the negative class is enormous.

What should I set contamination to?

Start with your best estimate of the true anomaly rate. If you genuinely do not know — which is common — set it to 'auto' and treat the resulting decision threshold as arbitrary, then choose your own threshold from the score distribution. Remember what contamination actually does: it does not change the model, it only sets where the cutoff falls. The underlying anomaly scores are identical whatever value you pass.

Should I use a supervised classifier instead?

If you have enough labelled anomalies — a few hundred at minimum — a supervised model with class weighting will usually outperform any unsupervised detector, because it learns what the anomalies actually look like rather than only what normal looks like. In practice, the right architecture is often a hybrid: an unsupervised detector to triage the enormous unlabelled volume, and a supervised model trained on the confirmed cases that humans have investigated. The unsupervised stage exists because labels are expensive, not because supervision is inferior.

My detector has great PR-AUC but the operators ignore it. Why?

Almost certainly alert volume. PR-AUC measures ranking quality across all thresholds; it says nothing about how many alerts you will actually generate per day. A detector with excellent PR-AUC operating at a threshold that fires 400 times a day will be ignored within a week — this is alert fatigue, and it is the most common reason anomaly detection systems fail in production. Always report the expected daily alert count alongside the quality metric, and pick a threshold that keeps the queue reviewable.