How a Learning Algorithm Works

01 Learning Objectives

02 Motivating Example

Suppose you want to predict how well a student will do on an exam based on how many hours they studied. You have data from three previous students:

Even without any formal method, the pattern is obvious: more study hours goes with a higher score. But how would a computer discover this pattern and use it to make predictions?

One answer would be to hand-write a rule: if hours is 2, predict 52; if hours is 5, predict 68; if hours is 8, predict 87. This is not a model — it is a lookup table, and it fails the moment a new student studies for 6 hours. The whole point of learning is to produce something that generalises beyond the examples it was trained on.

A better answer is to assume a relationship of a particular form — say, that the score is roughly a straight-line function of the hours studied — and then let the computer work out the exact line that fits best. That line becomes the model. This tutorial is about how the computer finds that line.

03 Concept Explanation

The four terms

Every machine learning problem can be described in terms of four things. Get these terms straight now and every algorithm you meet later will slot into the same framework.

Feature

An input variable. Something we measure about each example and give to the model. In the study-hours example, the feature is hours studied. In an image classifier, the features would be the pixel values. In a spam filter, the features might be the frequency of certain words.

Label

The output we want to predict. In supervised learning, every training example comes with a label — the correct answer we want the model to learn to reproduce. In the study-hours example, the label is exam score. In a spam filter, the label is spam or not spam.

Parameter

A number inside the model that the learning algorithm adjusts during training. Parameters are not given; they are discovered. In a straight-line model, the parameters are the slope and intercept. In a neural network, the parameters are the weights on every connection between neurons.

Loss

A single number that measures how wrong the model currently is. The loss is computed by comparing the model's predictions to the true labels. Small loss means the model is doing well; large loss means it is doing poorly. The entire purpose of training is to make the loss as small as possible.

Features and labels are given; parameters are learned

This is the distinction that trips up most beginners, so let us be precise about it.

Features and labels come from the data. They are what you have before training begins. A spreadsheet of historical data, a collection of labelled photographs, a database of past transactions — all of these provide features and labels.

Parameters do not come from the data. They are what the algorithm produces by looking at the data. Before training, the parameters have arbitrary initial values. After training, they have values that make the model fit the data well.

A useful way to think about it: features and labels are the question (here is some input, what is the correct output?), and parameters are the answer (here is the pattern that connects them).

The model as a function

A model is a function that takes features and parameters and produces a prediction. In mathematical notation:

prediction = model(features; parameters)

For a straight-line model with one feature, this function has a specific form:

prediction = w × feature + b

where w and b are the parameters. The algorithm's job is to find values of w and b that make predictions close to the true labels.

Every model in this course fits this same template. Linear regression (Module 2) is a straight line with more features. Logistic regression (Module 2) is a straight line passed through a function that squashes the output between 0 and 1. A decision tree (Module 3) has parameters that describe where the splits are. A neural network (Module 5) has thousands or millions of parameters. The forms differ; the template is always the same.

The training loop

Now we come to the core of this tutorial. The learning algorithm — whatever it is — is a loop with four steps:

1. Predict. Use the current parameter values to make a prediction for each training example.
2. Measure. Compare those predictions to the true labels and compute the loss. This gives a single number representing how wrong the current parameters are.
3. Adjust. Change the parameters slightly in the direction that reduces the loss.
4. Repeat. Go back to step 1 with the new parameter values.

That is the entire process. Every algorithm in this course runs this loop. The differences between algorithms are in how the model is represented, how the loss is defined, and how the parameters are adjusted — but the shape of the loop is always the same.

The loop terminates when the loss stops decreasing, or after a fixed number of iterations, or when some other stopping condition is met. At that point the parameters are considered “trained,” and the model is ready to make predictions on new data.

In pseudocode

To see the loop in its most stripped-down form, here it is as pseudocode:

initialise parameters to arbitrary values

repeat until loss stops decreasing:
    for each training example:
        prediction = model(features; parameters)
        loss       = measure_error(prediction, true_label)
    parameters = adjust(parameters, to_reduce_loss)

Those six lines describe the training process of every machine learning model, from a straight line to a large language model. The complexity lives inside the three functions — model, measure_error, and adjust — not in the loop itself.

How the parameters get adjusted

Step 3 of the loop — “adjust the parameters to reduce the loss” — is the part that requires calculus in practice, but the idea behind it is simple enough to grasp without any mathematics.

Imagine the loss as a landscape, with the parameters as coordinates. For a straight-line model, the loss landscape is a curved surface over the plane of possible w and b values. Some points on this surface are high (large loss) and some are low (small loss). Training means navigating from wherever you started to a low point.

The algorithm does this by working out which direction — up or down in w, up or down in b — would reduce the loss, and taking a small step in that direction. Then it repeats. Over many iterations, the parameters drift towards a region where the loss is small.

The size of the step matters. If steps are too small, training takes a very long time. If steps are too large, the algorithm may overshoot the low point and never settle. Finding the right step size — called the learning rate — is one of the practical skills of machine learning. We will return to it in Module 2.

04 Visual Illustration

The diagram below shows the study-hours data from the motivating example, with two candidate models drawn over it. The pale red line is the initial guess — a flat line that makes the same prediction regardless of how many hours were studied. The blue line is the model after training. The vertical segments show the residuals: how far each prediction is from the actual label.

05 Worked Example

Let us walk through two iterations of the training loop on the study-hours data. The purpose is not to teach the exact arithmetic of gradient descent — that comes in Module 2 — but to show what the loop actually computes at each step.

The trained model — score ≈ 6 × hours + 39 — is now the “knowledge” the algorithm has extracted from the data. It can predict the score of a new student who studied for six hours (predicted score: 6 × 6 + 39 = 75), even though no student in the training data studied for exactly six hours. This is what generalisation means, and it is the reason we train models rather than simply memorising the training data.

06 Interactive Activity

07 Self-Check Questions

Try each question before revealing the answer. If you get one wrong, re-read the section it refers to before continuing.

08 Practice Exercises

1. Identify the four terms. For each scenario, list the feature, label, parameters, and loss.
   • A model that predicts the fuel efficiency of a car from its weight and engine size.
   • A model that predicts whether an email is urgent or not, based on its subject line and sender.
   • A model that predicts a student's final grade from their midterm score and attendance rate.
2. Compute the loss. A model predicts the following values and receives these true labels. Compute the mean squared error (MSE).

   Prediction	True label
   3.2	3.0
   5.8	6.0
   4.1	4.5
   6.9	7.0

3. Order the training loop. Place the four steps of the training loop in the correct order: adjust parameters, measure the loss, predict, repeat.
4. True or false, with justification.
   • Features and labels are learned during training.
   • A model's parameters have values before training begins.
   • A loss of zero on the training data means the model is perfect.
   • The learning rate controls how many training examples the model sees.

09 Problem Sets

Analytical problems

1. The tutorial's worked example trained a straight-line model on three data points. Suppose we now add a fourth data point: (3, 45).

   (a) Would the trained parameters from the tutorial (w = 6, b = 39) still be optimal for the four points? (b) Sketch or describe what the new optimal line would look like. (c) Would the loss increase or decrease, and why?

2. Consider a model with zero parameters. Is it still a model in the sense described in this tutorial? What would its predictions look like, and would any learning take place?

3. The tutorial describes the loss as a landscape over the parameters, and training as navigation through that landscape. Describe, in words, what shape this landscape would have for a linear model with two parameters. What shape would it have for a very complex model with millions of parameters? Why does that shape matter for training?

4. The tutorial states that a model achieving zero loss on training data may still be a bad model. Give three distinct reasons why this might be true, each corresponding to a different failure mode from T-07.

Technical problems

1. Compute two iterations of the training loop by hand for the following problem. The model is prediction = w * x + b, starting with w = 0 and b = 0. The loss function is mean squared error. The training data is:

   x (feature)	y (label)
   1	2
   2	4
   3	6

   (a) Compute the loss with the initial parameters. (b) If w were increased to 1 and b kept at 0, would the loss go up or down? Verify by Computing (c) What are the optimal values of w and b? (d) Was the model able to reach the optimum in one step?

2. A model has the following performance at different parameter values:

   w	Loss
   0.0	15.2
   0.5	8.7
   1.0	3.4
   1.5	0.9
   2.0	0.4
   2.5	1.2
   3.0	3.8

   (a) What is the optimal value of w? (b) If the learning rate is too large, what might happen to the training process? Explain with reference to this table. (c) If the learning rate is too small, what is the consequence? (d) Sketch what a good learning rate would look like on this table.

3. Write pseudocode for a training loop that stops when the loss changes by less than 0.001 between iterations, or after a maximum of 1,000 iterations, whichever comes first. Your pseudocode should compute the loss on the full training set at each iteration and track the parameters that achieved the lowest loss.

Theoretical questions

1. Why does the training loop require a loss function rather than simply counting the number of incorrect predictions? What properties must a loss function have for the parameter adjustment step to work?
2. The tutorial describes the parameter adjustment step as “moving in the direction that reduces the loss.” What information does the algorithm need in order to determine this direction? Could a learning algorithm work without this information?
3. Is it possible for a model to have too many parameters for the amount of training data? What would happen if you tried to train such a model? Use the language of bias and variance in your answer.
4. The training loop is described as the same process for every machine learning algorithm. Is this really true, or does the description break down for some classes of algorithms? Consider decision trees, K-means, and support vector machines.
5. If training is fundamentally about reducing a loss function, and the loss function is chosen by the practitioner, what are the ethical implications of that choice? Can a perfectly trained model be ethically problematic because of how its loss was defined?

10 Discussion and Projects

Discussion questions

1. The loss function is chosen by the practitioner, not given by the problem. How much does this choice determine what the model will learn? Give an example where two different loss functions would produce very different models on the same data.
2. The tutorial describes the training process as navigation through a loss landscape. For very complex models, this landscape has billions of dimensions. Is the metaphor still useful at that scale, or does it break down?
3. If a model's parameters can be adjusted to fit any pattern in the training data, what prevents it from learning spurious correlations? Is there a way to build this protection into the training algorithm itself, or must it come from the data?
4. When a model achieves low loss, do we say it has “learned” the underlying pattern, or only that it has found parameters that make accurate predictions? Is there a meaningful difference between these two descriptions?

Case study: Netflix's recommendation algorithm

Critical-thinking activity: designing a loss function

Homework assignment

Mini-project: implement the training loop

11 Summary

12 Further Reading

The following sources expand on the mechanics covered in this tutorial.

• Google Developers. Machine Learning Crash Course, “Descending into ML” and “Reducing Loss” modules.
  A short, interactive introduction to loss and gradient descent. Strongly recommended if you want to see the ideas from this tutorial in a more visual format.
• Géron, A. (2022). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (3rd ed.). O'Reilly. Chapter 4.
  Chapter 4 covers training linear models in detail, including the mathematics of gradient descent. Best read after Module 2 of this course.
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 5 and Section 8.1.
  A rigorous treatment of the concepts in this tutorial from a mathematical perspective. Dense but excellent as a reference. The book is freely available online.
• 3Blue1Brown. “Gradient descent, how neural networks learn.” Neural Networks series.
  A visual, intuitive explanation of the training loop and how parameters are adjusted. Excellent at building the kind of intuition this tutorial aims for.