The Machine Learning Lifecycle

01 Learning Objectives

02 Motivating Example

Two teams at the same company are given the same task: build a model to predict which customers are about to cancel their subscription. Both teams have the same data, the same tools, and roughly the same level of skill.

Team A begins immediately. They take the customer database, pick a classification algorithm, split the data into training and test sets, and train a model. It achieves 91% accuracy. They deploy it. Within a month, the company notices that the model's predictions are not matching what the customer retention team is seeing. Half the customers the model flags as “about to leave” had just renewed their subscriptions. And the retention team is spending time on customers who were never going to cancel anyway.

Team B begins by asking questions. What does “about to cancel” actually mean — someone who has clicked a cancel button, or someone whose usage has dropped sharply, or someone whose contract is nearly up? What decision will the model support, and how much time does the retention team have to act on each prediction? What data exists about customers who did cancel, and how was that data collected? Which customer segments matter most?

After two weeks of this work, Team B discovers something Team A never noticed: the customer database contains a field called last_support_ticket that is only populated after a customer has contacted support to cancel. Team A had used this field as a feature. Their model was trivially accurate because it had found a field that leaked the outcome — a textbook case of the data leakage failure from T-07. Team B, having asked what each field actually meant and when it was recorded, excluded it. Their model achieves 78% accuracy, but it is measuring something real.

The difference between the two teams is not skill or effort. It is process. Team B followed the machine learning lifecycle; Team A skipped it.

03 Concept Explanation

The machine learning lifecycle is a structured process for building and maintaining machine learning systems. It has eight stages. Well-run projects move through them in roughly this order, but they also loop back: discovering a problem at stage 6 frequently sends a team back to stage 2.

Stage 1: Problem framing

Before touching data or algorithms, define precisely what problem the system is being asked to solve. This sounds obvious and is almost always the stage that teams rush through.

The key questions are: What decision will this model support? Who will act on its predictions, and what will they do differently? What does success look like, in measurable terms? What is the cost of being wrong?

In the churn example, Team B's questions were all framing questions. What does “about to cancel” mean? Which segments matter? How much time does the retention team have? Every one of these determines what data is needed, what the label should be, and how the model should be evaluated.

Framing problems are the most common cause of machine learning projects failing without ever technically failing. A model that accurately predicts the wrong thing is still useless.

Stage 2: Data collection

Once the problem is framed, you need data. Where will it come from? How much do you need? Is the data available at all, and if not, how will you obtain it?

This stage is often the most underestimated. Real datasets almost never exist in the form you need them. If you need labelled examples, someone has to label them — which is expensive, slow, and error-prone. If you need data from multiple sources, you have to reconcile them. If you need more data than exists, you may have to collect it yourself or decide the project is not viable.

Data collection is also where legal, ethical, and privacy considerations surface. Do you have the right to use this data? Have the people it describes consented? Will the way you collect it bias the results? These questions are not optional, and they are much easier to answer before collection begins than after.

Stage 3: Data preparation

Raw data is almost always messy. Missing values, inconsistent formats, duplicate records, outliers, and errors are the norm, not the exception. Data preparation means cleaning the data and shaping it into a form a model can learn from.

Typical activities include: handling missing values (drop, impute, or flag them); converting categorical variables into numerical form; normalising or standardising numeric features so they are on comparable scales; removing duplicate records; correcting errors; and splitting the data into training, validation, and test sets.

This stage is where a large fraction of the total project time is spent, and where many of the failure modes from T-07 originate. Data leakage usually happens here, when a feature is included that should not be. Spurious correlation often originates here, when a dataset is shaped in a way that accidentally encodes the wrong pattern.

Stage 4: Exploratory data analysis

Before training a model, look at the data. Compute basic statistics. Plot distributions. Check for correlations. Look for unexpected patterns or anomalies. This is the stage where you develop intuition about what the model will be able to learn.

Exploratory analysis is often the stage that reveals problems early. If the classes are extremely imbalanced, you want to know that before training. If a feature has an odd distribution, you want to know why. If two features are almost perfectly correlated, you may only need one of them.

It is also the stage that catches the husky/wolf problem. Plotting the data — and, ideally, looking at individual examples — is often how practitioners discover that the correlations the model will find are not the ones they intended.

Stage 5: Feature engineering and model selection

Now comes the stage most people think of as machine learning: choosing what the model should look at and which algorithm it should use.

Feature engineering is deciding which aspects of the raw data the model should use as inputs, and in what form. Sometimes this means selecting a subset of existing columns. Sometimes it means constructing new features — for example, converting a raw date into day-of-week, month-of-year, and is-holiday. Sometimes it means transforming a feature, such as taking the logarithm of a skewed numeric column.

Model selection is choosing the algorithm. Despite the attention it receives, this is usually the least consequential decision in the lifecycle. For many problems, several algorithms will give similar results, and the differences between them are small compared to differences in data quality or feature engineering. This is why the course spends two modules (Module 2 and Module 3) introducing algorithms and four modules on everything else.

Stage 6: Model training and evaluation

Train the chosen model on the training data, evaluate it on the validation data, and iterate. This is where the training loop from T-05 actually runs, and where the bias–variance tradeoff from T-07 becomes practically important.

Evaluation means much more than Computing accuracy. It means asking: does the model perform well on subgroups, or only on average? Is it calibrated — when it says 80% confident, is it right 80% of the time? Does its performance hold up under cross-validation? Does it meet the requirements established back at stage 1?

The most common reason a project loops backward from this stage is that the model performs well but does not answer the question the project was actually asked. That is not a modelling failure — it is a framing or feature engineering failure, and it sends the team back to an earlier stage.

Stage 7: Deployment

A model that stays in a notebook has no value. Deployment means integrating the model into whatever system will actually use it: a web application, a mobile app, a batch processing job, or an internal dashboard.

Deployment raises questions that do not arise during modelling. How fast does the prediction need to be? Does it need to run on every request, or can it be precomputed? How will the model's inputs be supplied at prediction time, and how do they differ from the training data? What happens if the model is unavailable — is there a fallback?

Many projects that succeed at modelling fail at deployment. The model is accurate but too slow; the inputs it needs are not available at serving time; the infrastructure team cannot support the framework it was built in. These are the concerns of Module 6, and they are just as important as the modelling itself.

Stage 8: Monitoring and maintenance

Once deployed, a model does not stay good forever. The world changes, and the model's assumptions gradually stop holding. Distribution shift, in the terms of T-07, is not a possibility but a certainty.

Monitoring means watching the model's behaviour in production: how often does it make predictions? How do its outputs compare to what was expected? Have the input distributions shifted? Are the outcomes still matching the predictions? Are downstream systems or users reporting problems?

Maintenance means acting on what monitoring reveals: retraining the model on fresh data, adjusting thresholds, replacing features that have become unreliable, or retiring the model entirely if the problem has changed beyond what it can address.

Monitoring and maintenance are not a single stage but an ongoing commitment. A deployed machine learning system is never finished; it is a living component that requires continued attention for as long as it is used.

Why the lifecycle is iterative

The eight stages are presented in order, but no real project moves through them in a straight line. Most projects loop backward multiple times. A few common examples:

• From evaluation back to problem framing. The model works, but it turns out to be answering the wrong question.
• From evaluation back to data collection. The model performs well on the training data but poorly on subgroups. More data for those subgroups is needed.
• From monitoring back to data preparation. A deployed model is behaving strangely, and the cause turns out to be a change in how the upstream data is formatted.
• From deployment back to problem framing. The model works technically, but nobody uses it, because the predictions are not actionable in the way the users actually work.

This is not a sign that the process has failed. It is what the process is for. Each loop back is a discovery that reduces the risk of a much larger failure later.

Where the time actually goes

One of the most surprising things for newcomers to machine learning is how little of a real project is spent on modelling. Across industry surveys, the rough distribution of effort is:

• Problem framing: 5–10%
• Data collection: 10–20%
• Data preparation and exploration: 30–40%
• Feature engineering: 10–20%
• Model training and evaluation: 5–15%
• Deployment: 10–15%
• Monitoring and maintenance: ongoing, often 10–20% per year of the system's life

The exact numbers vary by project, but the shape is consistent: data work dominates, and modelling is a small fraction. This is why the course spends Unit 2 on data fundamentals and Unit 3 on the Python ecosystem before introducing any algorithms.

04 Visual Illustration

The diagram below shows the eight stages arranged in a cycle rather than a line. The arrows between adjacent stages show the normal flow of work, but the dashed arrows across the centre indicate the most common loops back: from evaluation to earlier stages, and from monitoring back to the beginning.

05 Worked Example

Let us walk through the churn prediction project from the motivating example, this time as Team B would have approached it. The goal is to see what each stage actually involves, and where the value of following the process comes from.

Notice how little of this story involves modelling. Team B spent nine weeks on the project, of which one week was spent on feature engineering and model selection and one week on training and evaluation. The other seven weeks were spent on framing, data collection, data preparation, exploratory analysis, and deployment. And the process did not end when the model was deployed; it continues for as long as the system is in use.

06 Interactive Activity

07 Self-Check Questions

Try each question before revealing the answer. If you get one wrong, re-read the section it refers to before continuing.

08 Practice Exercises

1. Name the stage. For each activity, state which stage of the lifecycle it belongs to.
   • Meeting with the end users to understand how they will act on the model's predictions.
   • Adding a “was_missing” column to preserve information about missing values.
   • Plotting a histogram to check the distribution of a feature.
   • Setting up an alert that fires if the model's weekly prediction count drops by 20%.
   • Comparing logistic regression and gradient boosting on the same dataset.
2. Order the stages. Place the following stages in the correct order: deployment, problem framing, data collection, training and evaluation, exploratory analysis, monitoring, data preparation, feature engineering.
3. Identify the loop back. For each scenario, identify the stage where the problem was discovered and the stage the team should return to.
   • The model performs well overall but poorly on a specific subgroup.
   • The model is accurate but takes 8 seconds per prediction, which is too slow.
   • Monitoring shows the model's accuracy has dropped by 15% over six months.
4. Estimate the time. For a six-month machine learning project, roughly how many weeks would you expect to be spent on each stage? Justify your estimates with reference to the industry figures from this tutorial.

09 Problem Sets

Analytical problems

1. A financial services company wants to build a model to detect money laundering. Currently, compliance officers manually review 500 transactions per day, of which about 2% are eventually confirmed as suspicious.

   (a) What are the key questions the team should ask at the problem framing stage? (b) What data would they need to collect? (c) What are the two most important failure modes this system would be vulnerable to from T-07? (d) How would you evaluate whether the system is a success, given that the compliance officers are already reviewing transactions?

2. The tutorial describes nine weeks of work for the churn project, of which only two were spent on modelling. Is this a fair distribution? Argue either for or against the claim that the lifecycle spends too much time on non-modelling activities.

3. A team is building a system to predict equipment failure in a factory. There are 200 machines, each with 50 sensors. Sensors report readings every ten seconds. In the past three years, there have been 30 failures.

   (a) What are the biggest challenges at the data collection stage? (b) What stage of the lifecycle is likely to take the longest, and why? (c) How would you approach the class imbalance problem? (d) What stage would you be most concerned about, given the small number of positive examples?

4. The tutorial identifies five common loop-backs in the lifecycle. Identify three additional loop-backs not mentioned in the tutorial and describe a scenario that would trigger each one.

Technical problems

1. A team has built a model that performs well during development but has the following characteristics once deployed:

   • The model is 500MB in size, and inference takes 2 seconds per request.
   • The system receives 100 requests per second.
   • The model's inputs include several features that are computed by a batch job that runs nightly.

   (a) What specific deployment problems does this system face? (b) Which of these problems originate in the modelling stage and which in the deployment stage? (c) What changes would you recommend at each stage to address them?

2. Consider a model whose input distribution has drifted such that the mean of a key feature has shifted by 30% since training. Write pseudocode for a monitoring system that would detect this drift and trigger an alert. Your system should:

   • Track the distribution of each feature over time.
   • Compare the current distribution to the training distribution.
   • Trigger an alert when the divergence exceeds a threshold.
   • Avoid false alarms from normal fluctuations.

3. A team's model has the following performance metrics on a held-out test set:

   Metric	Overall	Group A	Group B
   Accuracy	0.86	0.91	0.72
   Precision	0.84	0.89	0.68
   Recall	0.79	0.86	0.61

   (a) Is this model ready for deployment? Justify your answer. (b) Which stage of the lifecycle would you return to, and why? (c) What additional data would you want to see before making a decision? (d) If the discrepancy between groups could not be resolved, would you deploy the model? What alternatives would you consider?

Theoretical questions

1. The tutorial presents the lifecycle as a framework for reducing risk. But some projects fail even when they follow the process carefully. What kinds of failures does the lifecycle not protect against?
2. The lifecycle stages are presented as discrete. In practice, real projects often work on multiple stages in parallel. What are the implications of this for how the process should be managed?
3. Problem framing is described as the most underrated stage. Why do you think this is? What incentives in a typical organisation push teams to skip or rush the framing stage?
4. The distribution of effort across stages is described as an average across industry. For which kinds of projects would this distribution be most different? Consider research projects, consumer products, and safety-critical systems.
5. The lifecycle includes monitoring and maintenance as the final stage, but monitoring and maintenance are ongoing. Is the “stage” framing appropriate, or does it misrepresent the ongoing nature of this work?

10 Discussion and Projects

Discussion questions

1. Team B in the churn example spent two weeks on problem framing before touching data. Team A began modelling immediately. In a business environment where time-to-delivery is often pressured, how would you justify the two weeks of framing to a manager?
2. The lifecycle suggests that most of the work is data-related. If this is true, why do you think machine learning is so often portrayed in popular media as being primarily about algorithms?
3. A deployed machine learning system requires ongoing maintenance. Who should be responsible for this maintenance, and how should it be funded? Is monitoring the same kind of work as initial development?
4. Some companies deploy models without rigorous evaluation, and rely on user feedback to identify problems. Is this acceptable for some applications but not others? Where would you draw the line?
5. The tutorial presents the lifecycle as a way to reduce risk. But following the lifecycle slows down deployment. How should teams balance the desire to ship quickly against the risk of doing it wrong?

Case study: the Zillow Offers failure

Critical-thinking activity: plan a project

Homework assignment

Mini-project: write a lifecycle playbook

11 Summary

12 Further Reading

The following sources expand on the lifecycle and process covered in this tutorial.

• Google. “MLOps: Continuous delivery and automation pipelines in machine learning.” Google Cloud Architecture Center.
  A detailed, industry-oriented walkthrough of the deployment and monitoring stages. Longer and more technical than this tutorial, and best read after you have finished the course's own modules on those topics.
• Ng, A. (2018). Machine Learning Yearning. Self-published (available online).
  A short, practical book about the process of building machine learning projects. Written for practitioners, but highly accessible to students. Particularly strong on problem framing and evaluation.
• Amershi, S., et al. (2019). “Software Engineering for Machine Learning: A Case Study.” Proceedings of ICSE 2019.
  A research paper that describes how machine learning development actually happens in practice, based on an investigation at Microsoft. Useful for understanding why the process is often messier than the textbook lifecycle suggests.
• Huyen, C. (2022). Designing Machine Learning Systems. O'Reilly.
  A book-length treatment of the full lifecycle, with detailed guidance on each stage. Particularly strong on deployment, monitoring, and the operational concerns that academic courses usually leave out.