Limitations and Failure Modes

01 Learning Objectives

02 Motivating Example

In 2016, a team of researchers published a paper demonstrating a technique for explaining the predictions of complex image classifiers. As part of their work, they trained a standard classifier on a well-known dataset: photographs of huskies and wolves, labelled by which animal was present.

The classifier achieved excellent accuracy. On the test set, it correctly identified the animal in almost every image. By any conventional measure, the model had learned to distinguish huskies from wolves.

Then the researchers applied their explanation technique to see what the model was actually looking at. On images classified as “wolf,” the highlighted regions of the image — the pixels the model considered most informative — were almost always in the background. Snow.

The model had not learned to recognise wolves. It had learned to recognise snow. In the training data, most photographs of wolves happened to be taken in snowy landscapes, and most photographs of huskies happened to be taken indoors or on grass. The model had found the easiest way to separate the two classes: was there snow in the picture?

This worked perfectly on the test set, because the test set had the same statistical property as the training set. But it would fail catastrophically on a photograph of a husky standing in snow, or a wolf photographed in a zoo. The model had not learned anything about wolves at all.

This is one of the most common and most insidious failure modes in machine learning. The model is not wrong in any technical sense — it achieved high accuracy on the data it was given. The failure is in the mismatch between what the model learned and what we assumed it had learned.

03 Concept Explanation

The husky-and-wolf example is one instance of a broader problem: machine learning models extract whatever patterns are useful for reducing loss on the training data, and those patterns are not always the patterns we intended. This section surveys seven specific failure modes that arise from this general issue.

1. Overfitting and underfitting

The most fundamental failure mode concerns the relationship between a model's complexity and its ability to generalise. Two failures are possible, and they pull in opposite directions.

Underfitting occurs when the model is too simple to capture the underlying pattern. A straight line fit to data that follows a curve will perform poorly on both the training data and new data. The model is not flexible enough to represent the true relationship, no matter how much data it is trained on.

Overfitting occurs when the model is too complex. A model with enough parameters can memorise the training data perfectly — including its noise and its accidental quirks — without learning anything general. Such a model performs excellently on training data and poorly on new data. This is what happened with the husky/wolf classifier: it was not a simple model, and it found a pattern (snow) that worked perfectly on the training set but reflected an accidental property of how the data had been collected.

The tension between these two failures is called the bias–variance tradeoff. Simple models have high bias (they make consistent errors because they cannot represent the true pattern) and low variance (they produce similar predictions regardless of which particular training sample they were given). Complex models have low bias but high variance (small changes in the training data produce large changes in the model). The goal is to find the level of complexity that balances the two.

The tradeoff is fundamental. There is no model that is both maximally simple and maximally accurate. Every practical machine learning decision involves choosing where on this spectrum to sit.

2. Spurious correlation

Spurious correlation is the name for what happened in the husky/wolf case: the model has learned a correlation that happens to exist in the training data but does not reflect a real causal relationship. The correlation between snow and wolves in the dataset was real — it was just accidental, a byproduct of how the photographs had been collected.

Spurious correlations are dangerous because they are invisible in aggregate metrics. The model's accuracy is high. Its loss is low. Everything looks correct until the model is deployed in a context where the spurious correlation no longer holds.

They are also extremely common. Any time a dataset is collected in a way that introduces accidental patterns — and every real dataset is collected in some specific way — there is a risk of the model learning those patterns instead of the intended ones. This is one of the reasons data collection and curation dominate the practical work of machine learning.

3. Data leakage

Data leakage occurs when information that would not be available at prediction time sneaks into the training data. The model learns to use that information, and reports excellent performance during training and testing, but fails completely in deployment because the leaked information is not available.

A classic example: suppose you are building a model to predict whether a patient will develop a particular disease, using historical medical records. If the record contains a field like “date of diagnosis,” and you include that field as a feature, the model will trivially learn to detect diagnosis from the diagnosis field. Its reported accuracy will be spectacular. But it will be useless, because at prediction time — for a patient who has not yet been diagnosed — that field is empty.

Leakage is one of the most common practical mistakes in machine learning projects. It is also one of the easiest to miss, because the failure is invisible until deployment. Careful practitioners always ask, for every feature: would this actually be available at the moment the prediction needs to be made?

4. Distribution shift

Machine learning models assume that the data they see in production will resemble the data they were trained on. When that assumption fails, the model's performance degrades. This is called distribution shift, and it comes in two main forms.

Data drift (also called covariate shift) occurs when the distribution of inputs changes but the relationship between inputs and outputs stays the same. If a fraud detection model was trained on transactions from before a major holiday season, and the transaction amounts during the holiday are much larger on average, the input distribution has shifted. The model's underlying assumptions may still be valid, but its calibration may be off.

Concept drift occurs when the relationship between inputs and outputs changes. A model trained to predict customer churn may become less accurate over time as the business changes its pricing, its product offerings, or its competitive position. The inputs might look the same, but the meaning of the inputs has changed.

Both forms of shift are unavoidable in any real deployment. The question is not whether a model will experience distribution shift, but how quickly it will degrade and how the degradation will be detected. This is why deployed models are monitored continuously, and why they are periodically retrained on fresh data.

5. Class imbalance

Many of the most valuable machine learning applications involve predicting rare events: fraud, disease, equipment failure, security breaches. When one class is much rarer than the other, standard training procedures fail in subtle ways.

The simplest failure is what T-06 described: a model that predicts the majority class for every input achieves high accuracy while being useless. This is why detection problems use metrics like precision and recall rather than accuracy.

But even with proper metrics, class imbalance is difficult. With very few positive examples, the model has limited signal to learn from. It may overfit to the specific positive examples it has seen. Standard remedies exist — resampling, class weighting, synthetic examples — but each introduces its own complications. Detecting rare events is fundamentally harder than detecting common ones.

6. Adversarial environments

Some machine learning applications operate against opponents who actively try to defeat them. Fraud detection is the classic case: fraudsters adapt their behaviour once they learn what the model flags. Spam filtering is another: spammers constantly develop new techniques to evade filters. Any system that makes decisions about people — loan approval, insurance pricing, content moderation — may face users who try to game it.

Adversarial environments violate a basic assumption of most machine learning: that the data distribution is independent of the model. When the model's own behaviour influences the data it will see next, standard training and evaluation procedures become unreliable. Concept drift, in particular, accelerates dramatically.

A special case is adversarial examples: inputs that have been deliberately crafted to fool a model. A photograph with a few pixels altered in a specific way can cause an image classifier to produce an entirely wrong prediction, even though the alteration is invisible to a human. Adversarial examples matter for security-sensitive applications, and they illustrate how differently a model sees the world compared to a human.

7. Feedback loops

When a model's predictions influence the data it will later be trained on, a feedback loop can develop. The model's biases become self-reinforcing over time.

Consider a predictive policing system that identifies neighbourhoods as high-risk and directs patrols there. The increased patrols produce more arrests in those neighbourhoods, which is recorded as data, which makes the model predict even higher risk for the same neighbourhoods. The model is not learning about crime; it is learning about where the police have been sent. But the data it is trained on cannot distinguish these two things.

Feedback loops are ethically fraught because they can entrench existing patterns of inequality. They are also technically difficult, because the causal structure of the problem is different from what the model assumes. This is one of the cases where the choice of algorithm is irrelevant — the failure is in what the system is being asked to predict, not in how it is being built.

The common thread

Look across these seven failure modes and a pattern emerges. None of them is an algorithm problem. Overfitting is a complexity problem. Spurious correlation is a data collection problem. Data leakage is a feature engineering problem. Distribution shift is a deployment problem. Class imbalance is a problem with the structure of the phenomenon being predicted. Adversarial environments and feedback loops are problems with how the system interacts with the world.

This is the central insight of practical machine learning: failures almost never come from the algorithm. They come from the data, the deployment, and the assumptions we bring to the problem. This is why machine learning projects spend far more time on data and integration than on modelling, and why the lifecycle you will meet in T-08 is structured the way it is.

04 Visual Illustration

The diagram below shows the classic relationship between model complexity and error. The pale blue curve is the error on the training data, which always decreases as the model becomes more complex. The solid blue curve is the error on new data the model has never seen. Its shape is what makes overfitting visible: it starts high (underfitting), reaches a minimum, and then rises again as the model begins to memorise the training data.

05 Worked Example

Let us return to the husky-and-wolf classifier from the motivating example and walk through a systematic diagnosis. The purpose is to show how the seven failure modes are used in practice: not as a checklist to memorise, but as a set of questions that reveal what has gone wrong.

This diagnosis illustrates the central practical lesson of this tutorial. When a machine learning model fails, the productive question is rarely “which algorithm should we try instead?” and almost always “what is the model actually learning from, and why?” The failure modes in this tutorial are diagnostic categories that help answer that question systematically.

06 Interactive Activity

07 Self-Check Questions

Try each question before revealing the answer. If you get one wrong, re-read the section it refers to before continuing.

08 Practice Exercises

1. Name the failure mode. For each scenario, identify the most likely failure mode from the seven categories in this tutorial.
   • A model that performs perfectly on training data but poorly on test data.
   • A model that performs well on test data but fails in production because a feature is not available at prediction time.
   • A model that performs well for two years but gradually becomes less accurate.
   • A model that predicts the majority class for every input.
   • A model that learns that snow predicts wolves.
2. Classify the shift. For each scenario, state whether it is data drift or concept drift.
   • A fraud detection model becomes less accurate after a new payment method is introduced.
   • A movie recommendation model becomes less accurate after users' tastes change during a pandemic.
   • A medical diagnosis model becomes less accurate because a new imaging protocol produces slightly different images.
3. Match the remedy. Match each failure mode with the most appropriate remedy.

   Failure mode	Remedy
   Overfitting	Monitor deployed model and retrain periodically
   Data leakage	Reduce model complexity or add regularisation
   Distribution shift	Remove features not available at prediction time
   Class imbalance	Use precision and recall instead of accuracy

4. True or false, with justification.
   • If a model achieves high accuracy on a validation set, it will perform well in production.
   • A model trained on data with 99% positive examples will always predict the majority class.
   • Distribution shift is avoidable if the training data is large enough.

09 Problem Sets

Analytical problems

1. Consider a model that achieves 95% accuracy on the training set and 72% accuracy on a validation set. Another model achieves 78% on both. Which model would you choose for deployment, and why?

   (a) Which failure mode does the first model exhibit? (b) What would you do to address this failure mode? (c) Would your answer change if the validation set were very small?

2. The tutorial describes the husky/wolf classifier as a case of spurious correlation rather than overfitting. Explain in detail why this distinction matters, and why the remedies for the two failures are different.

3. A hospital builds a model to predict which patients will develop sepsis. The model is trained on data from 2018–2020 and deployed in 2021. By 2023, its performance has degraded significantly.

   (a) What are the possible causes of the degradation? (b) How would you distinguish between these causes in practice? (c) Would the remedy be the same for all causes, or would different causes require different responses?

4. A predictive policing system directs patrols to neighbourhoods that the model identifies as high-risk. Over time, arrests in those neighbourhoods increase, which the model interprets as further evidence of high risk. Identify the failure mode at work, and explain why simply retraining the model will not fix it.

5. The tutorial describes seven failure modes. In practice, a single failed system often exhibits more than one of these simultaneously. Describe a plausible scenario in which three or more failure modes co-occur, and explain how they interact.

Technical problems

1. A model is trained to predict whether a loan application should be approved. The training data has the following properties:

   • 90% of applications were approved.
   • The feature days_since_application is included.
   • The feature loan_officer_notes is included, and these notes are only entered after the decision has been made.

   (a) Which of these features constitutes data leakage? (b) Why would the model achieve suspiciously good performance in testing if the leaked feature were included? (c) What would happen in deployment? (d) What other failure modes might this model be vulnerable to, given the class distribution?

2. A model achieves the following performance across two years:

   Period	Precision	Recall	F1
   Q1 (deployment)	0.82	0.75	0.78
   Q2	0.81	0.74	0.77
   Q3	0.78	0.70	0.74
   Q4	0.72	0.63	0.67
   Q5	0.65	0.55	0.60

   (a) What pattern does this data show? (b) Which of the seven failure modes is most likely? (c) If you were the team, at what point would you have started investigating? What would you have done? (d) Is there a point at which you would take the model offline entirely? Justify your answer.

3. A team trains a decision tree on a small dataset. They vary the maximum depth of the tree and record the training and validation error:

   Max depth	Training error	Validation error
   1	0.42	0.44
   2	0.31	0.34
   3	0.22	0.28
   5	0.12	0.27
   8	0.05	0.31
   12	0.01	0.38
   20	0.00	0.46

   (a) At which depth is the model optimally complex? (b) At which depth does overfitting begin? (c) What does the depth-20 model tell you about the training data? (d) What technique could you use to grow a tree to depth 20 without overfitting?

Theoretical questions

1. Bias and variance are often described as pulling in opposite directions. Is it possible for a model to have both high bias and high variance? If not, why not? If so, what would such a model look like?
2. The tutorial claims that failures are never algorithmic. Is this literally true, or is it a useful simplification? Are there any genuine failure modes that originate in the algorithm itself?
3. Data leakage and spurious correlation both involve the model exploiting information that should not help. What distinguishes them, and why do the remedies differ?
4. Some machine learning practitioners argue that the best defence against all failure modes is to collect more data. Is this true? For which of the seven failure modes would more data help, and for which would it not?
5. The tutorial frames the bias–variance tradeoff as a fundamental tension. Is it really fundamental, or is it a consequence of the specific training methods currently used? Could a future method avoid the tradeoff entirely?

10 Discussion and Projects

Discussion questions

1. Many machine learning failures are invisible until deployment. What responsibility do practitioners have to anticipate failures that they cannot observe during development?
2. The tutorial argues that failures are usually data problems, not algorithm problems. If this is true, why do so many courses and textbooks focus predominantly on algorithms?
3. Feedback loops are described as ethically fraught because they can entrench inequality. Should systems that risk feedback loops be subject to stricter regulation than other machine learning systems?
4. The husky/wolf classifier was technically working as designed. Does this mean that the system was not at fault? Who should be responsible when a system fails in the way it was designed to?
5. Some researchers argue that interpretable models are inherently safer because their failures are visible. Others argue that interpretable models are less accurate and therefore less safe. Where do you stand?

Case study: COMPAS and the limits of fair machine learning

Critical-thinking activity: auditing a dataset for failure modes

Homework assignment

Mini-project: build a failure-mode audit tool

11 Summary

12 Further Reading

The following sources expand on the failure modes covered in this tutorial.

• Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). ““Why Should I Trust You?”: Explaining the Predictions of Any Classifier.” Proceedings of KDD 2016.
  The paper where the husky/wolf example originates. It introduces LIME, a technique for explaining individual predictions, and the husky/wolf case is used to motivate why such techniques are needed.
• Sculley, D., et al. (2015). “Hidden Technical Debt in Machine Learning Systems.” Advances in Neural Information Processing Systems, 28.
  Recommended reading for anyone building real ML systems. It catalogs the maintenance and integration problems that make deployed machine learning harder than the algorithms suggest.
• Goodfellow, I., McDaniel, P., & Papernot, N. (2018). “Making machine learning robust against adversarial inputs.” Communications of the ACM, 61(7), 56–66.
  A readable introduction to adversarial examples and why they matter for security-critical applications.
• Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and Machine Learning. MIT Press (available online).
  A book-length treatment of how models can perpetuate and amplify inequality. The feedback loop problem in this tutorial is developed at length. Also directly relevant to Module 6.