COMP328 Fundamentals of Machine Learning

Unit 4 · Tutorial 4 of 8

The Cost Function and the Loss Surface

In Tutorial 3 you found the least-squares solution by solving the normal equations. That worked because linear regression has a closed form. Almost nothing else in machine learning does. This tutorial reframes fitting as a search over a landscape — and shows that for linear regression the landscape is a bowl with exactly one bottom.

1 Learning objectives

By the end of this tutorial you will be able to

  • Distinguish a loss function (one example) from a cost function (the whole dataset).
  • Convert RSS into mean squared error and explain three reasons the division by n matters.
  • Describe the loss surface — what its axes are, what its height means, and how to read a contour plot of it.
  • Define convexity, verify that the MSE cost function is convex, and explain why this is the single most valuable property a cost function can have.
  • Compute the gradient of MSE with respect to the parameters, and interpret it as the direction of steepest ascent.
  • Explain how feature scaling changes the shape of the loss surface, and why that matters for iterative fitting.
  • Sketch the path that gradient descent takes on a convex surface, and explain why it can zig-zag even when there is only one minimum.

Estimated time

30 minutes of reading, plus 20–25 minutes for the practice exercises and problem set. The visualisations in Section 4 carry a lot of the intuition — look at them before reading Section 3.4.

2 Motivating example

Tutorial 3 left you with a formula. Given the sums Sxy and Sxx, the least-squares slope is Sxy/Sxx and that is that. So why not move on to the next algorithm?

Because the formula only works here. Logistic regression — the very next unit — has no closed-form solution. Decision trees, neural networks, support vector machines, and essentially every model that has been built since the 1970s: no closed-form solution. If the only fitting technique you possess is "solve the normal equations", you can fit exactly one model.

So this tutorial does something that looks at first like a step backwards. Instead of solving for the best parameters, we will search for them — the way you would search for the lowest point in a foggy valley, by feeling the slope of the ground under your feet and stepping downhill. It is slower than the formula and it needs a starting point and a step size. But it works when no formula exists, which is almost always.

Before you can walk downhill, you need a landscape. That is what this tutorial builds.

The question we are answering

For every possible pair of parameters (w, b), how bad is the resulting line? If we could plot badness against every possible parameter choice, what would that picture look like?

Why it matters beyond Unit 4

Every model you train for the rest of this course is fitted by minimising some cost function. The vocabulary introduced here — loss surface, convexity, gradient, learning rate, conditioning — is the shared language of all of it.

The valley metaphor, taken seriously

Imagine standing in a foggy valley and wanting to reach the lowest point. You cannot see more than a metre. But you can feel which way the ground slopes. So you take a small step in the downhill direction, feel again, and repeat. If the valley has one basin, you will reach the bottom. If it has many, you will reach a bottom — and it may not be the deepest one.

That metaphor is not decoration. It is a precise description of gradient descent, and the "one basin or many" question is exactly the question of convexity.

3 Concept explanation

3.1 From RSS to mean squared error

In Tutorial 3 the criterion was the residual sum of squares:

RSS(w, b) = Σi=1n (yi − w xi − b)²

The least-squares solution minimises this. But RSS has an awkward property: it grows with the size of the dataset. Fit the same model to 100 examples and to 100,000 examples drawn from the same process, and the RSS will be roughly a thousand times larger in the second case — even though the model is no worse.

Dividing by n fixes this. The result is the mean squared error:

Mean squared error (MSE) cost function

J(w, b) = (1/n) Σi=1n (yi − w xi − b)²

J is used throughout this course for the cost function. Other common names in the literature are C, L, or 𝒥. The choice of letter is arbitrary; the meaning is not.

Minimising J and minimising RSS give exactly the same parameters, because dividing by the positive constant n does not move the location of the minimum. It only rescales the vertical axis of the picture we are about to draw.

3.2 Why divide by n?

Three reasons, in increasing order of importance:

What dividing by n buys you
Reason Explanation
Comparability MSE is on the same scale regardless of dataset size, so you can compare the fit quality of two models trained on different amounts of data — and compare training error against validation error computed on a differently sized split.
Interpretability MSE is the average squared error per prediction, in squared units of the target. With rentals, MSE is measured in "rentals squared" — awkward, but the square root (RMSE) is in rentals, which is directly meaningful.
Mathematical convenience The 1/n cancels cleanly when you differentiate, and it makes the gradient an average rather than a sum — which means stochastic and mini-batch gradient descent (Module 5) are natural extensions rather than ad-hoc fixes.

3.3 Loss versus cost versus objective

These three words are used loosely in practice, but they name three distinct things, and the distinction becomes important when you read papers.

Loss function
A measure of error for one example. For linear regression, the squared error ℓ(w, b; xi, yi) = (yi − ŷi)².
Cost function
A measure of error for the whole training set, usually the average of the per-example losses. MSE is the cost function for linear regression.
Objective function
The thing you actually optimise. It is usually the cost function, but it may include extra terms — a regularisation penalty, for instance. In Module 3 you will meet ridge regression, whose objective is MSE plus a penalty on the size of the weights.

So the loss is per-example, the cost is the aggregate, and the objective is whatever you hand to the optimiser. In this unit they coincide closely enough that the distinction is a formality — but it is a formality that will save you confusion later.

3.4 The loss surface

Here is the conceptual leap of this tutorial. J(w, b) is a function of the parameters, not of the data. The data is fixed; we are varying the model.

That means we can plot it:

  • The horizontal axes are the parameters — one axis per parameter. For simple linear regression there are two, w and b, so the plot lives in three dimensions.
  • The vertical axis is the cost J: how badly that particular parameter setting fits the training data.
  • Each point on the surface corresponds to one complete model. A point high on the surface is a bad model; the lowest point is the best one.

The shape of this surface is called the loss surface or cost surface. Training a model means finding the lowest point on it.

Reading a contour plot

Three-dimensional surfaces are hard to read on a screen, so we usually flatten them. A contour plot draws lines of equal cost, exactly like a topographic map. Each closed curve is a set of parameter pairs that produce identical MSE. The innermost ring surrounds the minimum.

Two things to read from a contour plot: the location of the centre (where the best parameters are) and the shape of the rings (how easy the surface is to navigate). Circular rings mean an easy surface; long narrow ellipses mean a hard one.

3.5 Convexity

Now the crucial property. A function is convex if the line segment joining any two points on its graph lies on or above the graph. Informally: the surface is shaped like a bowl, never like a mountain range with multiple valleys.

Two consequences follow immediately, and they are the reason convexity is prized:

  1. Any local minimum is a global minimum. There are no false bottoms to get trapped in. If you walk downhill and stop, you have arrived at the best possible answer.
  2. The minimum is unique (assuming the features are not perfectly collinear). There is exactly one best parameter vector, and every correct algorithm will find it.

Is the MSE cost function for linear regression convex? Yes. Expanding the square shows that J is a quadratic function of the parameters, with a positive coefficient on the squared terms: 

J(w, b) = (1/n) Σ ( yᵢ − w xᵢ − b )²

        = (1/n) Σ ( yᵢ²  −  2 yᵢ(w xᵢ + b)  +  (w xᵢ + b)² )

        = (1/n) Σ yᵢ²  −  (2/n) w Σ xᵢyᵢ  −  (2/n) b Σ yᵢ
                       +  (1/n) w² Σ xᵢ²  +  (2/n) wb Σ xᵢ  +  (1/n) n b²

Every parameter appears at most squared, and the coefficients on w² and b² — namely (1/n)Σxi² and 1 — are positive. That is precisely the algebraic signature of a convex quadratic. Geometrically, the surface is an upward-opening paraboloid: a bowl.

Convexity is not guaranteed in general

Linear regression is convex. Logistic regression, surprisingly, is also convex. But neural networks are not convex — their loss surfaces genuinely have many local minima, plateaus and saddle points. This is why Module 5 spends considerable time on optimisation. Everything you learn about the convex case here is the baseline against which the hard cases are understood.

3.6 The gradient

If the loss surface is a landscape, the gradient is the compass. The gradient of J is the vector of its partial derivatives with respect to each parameter:

∇J = ( ∂J/∂w , ∂J/∂b )

For MSE, differentiating with respect to each parameter in turn:

Gradient of the MSE cost function

∂J/∂w = −(2/n) Σ xi (yi − ŷi) = −(2/n) Σ xi ei

∂J/∂b = −(2/n) Σ (yi − ŷi) = −(2/n) Σ ei

Two things to notice about these expressions:

  • Each derivative is proportional to a residual-weighted sum. The derivative with respect to w weights each residual by its feature value xi; the derivative with respect to b weights them all equally. That makes sense: w controls how steeply the prediction changes with the input, so an error on a large-x example is more informative about w than an error on a small-x example.
  • Both derivatives are zero exactly when the normal equations hold. Setting ∂J/∂b = 0 gives Σei = 0, and setting ∂J/∂w = 0 gives Σxiei = 0. These are the two normal equations from Tutorial 3. The calculus and the algebra are the same statement.

The gradient has a geometric meaning that you should internalise: it points in the direction of steepest ascent. Stand at a point on the loss surface and the gradient tells you which way is most sharply uphill. Which means the negative gradient points most sharply downhill — and that is the direction a minimiser wants to move.

3.7 The shape of the surface depends on the data

The loss surface is not a property of the algorithm. It is a property of the data. Change the data and you change the landscape — sometimes dramatically.

The most important way this happens in practice is through feature scaling. Suppose you have two features in a multiple regression: one measured in metres, ranging from 0 to 3, and one measured in millimetres, ranging from 0 to 3000. A one-unit change in the first is a large physical change; a one-unit change in the second is barely anything. The cost surface becomes extremely elongated along one axis, forming a long narrow valley rather than a round bowl.

Two surfaces, same data, different scaling
Surface shape Contours Gradient descent behaviour
Well-conditioned Nearly circular Moves almost directly to the minimum. Large learning rates are safe.
Ill-conditioned Long narrow ellipses Zig-zags back and forth across the narrow valley, making slow progress along it. Requires a small learning rate to avoid diverging.

This is the practical reason StandardScaler exists. Rescaling the features so they have comparable ranges makes the contours rounder, which makes iterative fitting faster and more stable. Tutorial 5 will show the effect numerically; the point here is that it is a property of the landscape, visible before you run a single optimisation step.

Zig-zag is not a bug

On a long narrow valley, the steepest direction at any point is nearly perpendicular to the valley floor. So gradient descent keeps crossing the valley and bouncing off the sides rather than heading straight down it. This happens on a perfectly convex surface with a single minimum — it has nothing to do with local minima. It is purely a consequence of the shape of the bowl. If you see a zig-zagging loss curve, the fix is rescaling the features or tuning the learning rate, not switching to a different algorithm.

3.8 What convexity does and does not buy you

Convexity is a strong guarantee, so it is worth being precise about its scope.

Guaranteed

  • No local minima other than the global one.
  • No saddle points that trap a minimiser.
  • Any algorithm that reaches a stationary point has found the optimum.
  • The optimum is unique if the features are linearly independent.

Not guaranteed

  • That gradient descent will reach the minimum in finite time — it may only approach it asymptotically.
  • That it will reach it quickly — the conditioning of the surface still matters.
  • That the minimum is a good model — a convex objective can still be the wrong objective for your problem.
  • Anything about generalisation. Training error and test error are different questions.

That third point deserves emphasis. You can minimise MSE perfectly and still have a model that is useless, because MSE may not be the quantity you actually care about. Tutorial 6 takes this up in detail.

4 Visual illustration

Cross-sections of a convex and a non-convex cost function Two panels. The left panel shows a convex parabola with a single global minimum, which is the shape of the mean squared error surface for linear regression. The right panel shows a wavy non-convex curve with two local minima and one global minimum, which is the shape encountered in neural networks. Convex — linear regression unique minimum parameter w cost J one basin only — safe to walk downhill Non-convex — neural networks local min A local min B local max parameter w several basins — where you start matters
Figure 1. A one-dimensional slice through two kinds of cost function. On the left, the convex MSE surface for linear regression: a single basin, so any downhill walk ends at the same place. On the right, a non-convex surface of the kind encountered in neural networks: two local minima, and a gradient-based method will settle into whichever basin it happens to start in.
Contour plot of the MSE loss surface with a gradient descent path Concentric elliptical contours of the mean squared error plotted in the space of slope and intercept. A zig-zagging path of dots descends from the upper right of the plot toward the centre, where the unique minimum lies. An arrow at the starting point shows the direction of the negative gradient. Slope w Intercept b descent path −∇J direction minimum
Figure 2. Contour lines of J(w, b) for the bike-share data, with a gradient descent path overlaid. Each ring is a set of parameter pairs producing identical MSE; the innermost ring encircles the least-squares solution. The path zig-zags because the valley is elongated: the steepest direction at each point is across the valley rather than along it.
Comparison of well-conditioned and ill-conditioned loss surfaces Two contour plots side by side. On the left, circular contours produce a gradient descent path that moves almost directly to the centre. On the right, elongated elliptical contours produce a path that zig-zags many times before reaching the centre. Well-conditioned nearly straight path — fast Ill-conditioned zig-zag path — slow, needs small steps Same data, same minimum — only the feature scaling differs
Figure 3. Conditioning is a property of the surface, not the algorithm. On the left, circular contours let gradient descent head almost straight for the minimum. On the right, the same problem with unscaled features produces long narrow ellipses, and the optimiser bounces between the valley walls instead of walking down the floor. Both surfaces are convex and both have the same unique minimum.

5 Worked example

We continue with the bike-share data from Tutorials 1 and 3, and this time we evaluate the cost function at several parameter settings, then compute the gradient at one of them.

Step 1 — Evaluate J(w, b) at four candidate models

Recall the eight observations: x = [8, 12, 15, 18, 21, 24, 27, 30] °C, y = [120, 180, 240, 300, 360, 430, 510, 560] rentals, and n = 8. From Tutorial 3, the least-squares solution is ŵ ≈ 20.65 and b̂ ≈ −62.60.

Below, each candidate pair is scored by Computing all eight predictions, all eight residuals, squaring them, summing, and dividing by 8.

MSE at four parameter settings
w b Predictions at the eight temperatures RSS MSE = RSS/8
20.0 0.0 160, 240, 300, 360, 420, 480, 540, 600 21,000.0 2,625.00
20.0 −50.0 110, 190, 250, 310, 370, 430, 490, 550 1,000.0 125.00
20.5 −60.0 104, 186, 247.5, 309, 370.5, 432, 493.5, 555 840.75 105.09
20.65 −62.60 102.6, 185.2, 247.2, 309.1, 371.1, 433.0, 495.0, 556.9 830.9 103.87
Least-squares solution (from Tutorial 3) ≈831 ≈103.9

The numbers tell the story of the surface. Moving from (20, 0) to (20, −50) cuts the cost by a factor of 21 — a huge improvement from a small change in the intercept. From there to (20.5, −60) it falls another 16%. From there to the optimum it falls only 1.2%. The surface is steep far from the minimum and almost flat near it.

That flattening is general, and it is why optimisation converges quickly at first and then slows down. It is also why you should not judge progress by the size of the improvement in the last few iterations.

Step 2 — Compute the gradient at (w, b) = (21, −70)

Start from a point that is near the minimum but not at it. Compute the predictions and residuals:

Residuals at w = 21, b = −70
x y ŷ = 21x − 70 e = y − ŷ x · e e²
812098+22176484
12180182−2−244
15240245−5−7525
18300308−8−14464
21360371−11−231121
24430434−4−9616
27510497+13351169
30560560000
Sum +5 −43 883

Applying the gradient formulas:

∂J/∂w = −(2/n) Σ xᵢ eᵢ
      = −(2/8) × (−43)
      = +10.75

∂J/∂b = −(2/n) Σ eᵢ
      = −(2/8) × (+5)
      = −1.25

∇J(21, −70)  =  ( +10.75 , −1.25 )

Step 3 — Interpret the gradient

The gradient is (+10.75, −1.25). Both components are small relative to the MSE values we computed earlier, which already tells us we are close to the bottom — recall that at (20, 0) the gradient with respect to w was around +1,872, roughly 175 times larger.

Reading the signs:

  • ∂J/∂w is positive, so increasing the slope slightly would increase the cost. To reduce the cost we should decrease the slope. And indeed, the optimum slope is 20.65, slightly less than the current 21.
  • ∂J/∂b is negative, so decreasing the intercept slightly would increase the cost. To reduce the cost we should increase the intercept. The optimum is −62.60, which is larger than the current −70.

So the direction of steepest descent, −∇J = (−10.75, +1.25), points toward a smaller slope and a larger intercept. That is the right general direction. Check it explicitly with a small step, using a learning rate α = 0.001: 

w_new = 21    − 0.001 × 10.75 = 20.98925
b_new = −70   − 0.001 × (−1.25) = −69.99875

MSE(21, −70)        = 883 / 8        = 110.375
MSE(20.98925, −69.99875)             ≈ 110.370

The cost decreases, by a tiny amount. Repeating this step many times, with a suitable learning rate, is gradient descent — and it is the subject of Tutorial 5.

Step 4 — Verify the gradient vanishes at the optimum

At the least-squares solution the two normal equations hold, which means both partial derivatives are zero: 

At w = 20.6502, b = −62.5976:
  Σ eᵢ ≈ 0        →  ∂J/∂b = −(2/8) × 0    = 0
  Σ xᵢeᵢ ≈ 0      →  ∂J/∂w = −(2/8) × 0    = 0

The gradient is the zero vector, so the surface is flat underfoot. This is the signature of a stationary point. Because the surface is convex, a stationary point must be the global minimum — there is nowhere else for it to be.

The same computation in code

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "temp_c":  [ 8, 12, 15, 18, 21, 24, 27, 30],
    "rentals": [120, 180, 240, 300, 360, 430, 510, 560],
})

x = df["temp_c"].to_numpy()
y = df["rentals"].to_numpy()
n = len(x)


def mse(w, b):
    """Mean squared error for the line y = w*x + b."""
    return ((y - (w * x + b)) ** 2).mean()


def gradient(w, b):
    """Gradient of MSE with respect to (w, b)."""
    e = y - (w * x + b)
    return -2 * (x * e).mean(), -2 * e.mean()


print(f"{'w':>8} {'b':>9} {'MSE':>12}")
for w, b in [(20.0, 0.0), (20.0, -50.0), (20.5, -60.0), (21.0, -70.0)]:
    print(f"{w:8.2f} {b:9.2f} {mse(w, b):12.2f}")

print(f"\nleast-squares optimum: MSE = {mse(20.6502, -62.5976):.2f}")

gw, gb = gradient(21.0, -70.0)
print(f"\ngradient at (21, -70) : dJ/dw = {gw:+.2f}, dJ/db = {gb:+.2f}")

gw0, gb0 = gradient(20.0, 0.0)
print(f"gradient at (20,   0) : dJ/dw = {gw0:+.2f}, dJ/db = {gb0:+.2f}")

gwo, gbo = gradient(20.6502, -62.5976)
print(f"gradient at optimum   : dJ/dw = {gwo:+.4f}, dJ/db = {gbo:+.4f}")
       w         b          MSE
   20.00      0.00      2625.00
   20.00    -50.00       125.00
   20.50    -60.00       105.09
   21.00    -70.00       110.38

least-squares optimum: MSE = 103.87

gradient at (21, -70) : dJ/dw = +10.75, dJ/db = -1.25
gradient at (20,   0) : dJ/dw = +1872.50, dJ/db = +100.00
gradient at optimum   : dJ/dw = -0.0000, dJ/db = +0.0000

Compare the two gradients. At (20, 0), far from the optimum, the gradient components are in the thousands. At (21, −70), close to the optimum, they are in the single digits. At the optimum itself they are zero to four decimal places. The magnitude of the gradient is a direct measure of how far you are from the bottom — which is exactly what makes it useful as a stopping criterion.

nb-04 · Exploring the loss surface Compute MSE over a grid of (w, b) values, draw the contour plot and the 3D surface, then overlay gradient descent paths for two different learning rates.

6 Interactive activity

Draw the landscape, then walk on it

In the Tutorial 4 notebook, build the loss surface for the bike-share data and explore it systematically. Work through the following and record what you find:

  1. Build the grid. Evaluate mse(w, b) for every combination of w in np.linspace(15, 26, 120) and b in np.linspace(-120, 20, 120). Store the results in a 120 × 120 array.
  2. Plot the contours. Use plt.contour(W, B, J, levels=30). Mark the least-squares solution with a star. Are the rings circular or elongated? What does that tell you?
  3. Confirm the shape. Slice the surface along b = −62.6 (holding w at the optimum intercept value) and plot J against w. Is it a parabola? Now slice along w = 20.65. Is that a parabola too?
  4. Walk downhill. Starting from (21, −70), run gradient descent for 200 iterations with a learning rate of 0.0005. Plot the path on top of the contours. How many steps does it take to get within 1% of the optimal MSE?
  5. Change the step size. Repeat with learning rates of 0.0001, 0.001 and 0.005. Which ones converge? Which overshoot? What happens at α = 0.01?
  6. Change the scaling. Now convert temperature to Fahrenheit (xF = 1.8xC + 32) and rebuild the surface. Compare the contour shapes before and after. Run gradient descent with the same learning rates and compare the convergence.
Show the discussion
  1. Grid and storage. Use broadcasting: W, B = np.meshgrid(w_vals, b_vals) and then compute J as a 2-D array. Doing this with a loop over 14,400 points is slow; the vectorised version is instant.
  2. Elongated rings. The contours are noticeably elongated, tilted ellipses rather than circles. That means the surface is ill-conditioned: the cost changes much more rapidly in some directions than others. This is why gradient descent zig-zags rather than heading straight for the centre.
  3. Both slices are parabolas. That is the definition of a convex quadratic in two variables — any one-dimensional slice through it is an upward-opening parabola. The curvatures of the two parabolas differ, which is precisely what the elongated contours were telling you.
  4. Convergence. With α = 0.0005, several hundred iterations are needed to get within 1% of the optimal MSE. The path curves in a spiral rather than travelling straight to the centre, reflecting the elongated shape of the contours. Notice that the first few iterations make enormous progress and the later ones barely move — the surface flattens near the bottom.
  5. Learning rates. α = 0.0001 converges but takes thousands of iterations. α = 0.0005 is workable. α = 0.001 is faster but starting to oscillate. α = 0.005 overshoots and diverges — the cost grows without bound until it overflows. α = 0.01 diverges immediately. The relationship between the largest safe learning rate and the curvature of the surface is the single most important practical fact about gradient descent.
  6. Scaling. In Fahrenheit, the optimal slope becomes approximately 20.65/1.8 ≈ 11.47 and the intercept changes accordingly, but crucially the contours become rounder — or more elongated, depending on which direction you scaled. What matters is the ratio of the curvatures, not the absolute values. The largest safe learning rate changes when the scaling changes, which is why you must always retune it after rescaling features.

The two numbers that govern everything

The behaviour of gradient descent on a quadratic surface is governed by the curvature in each direction. The largest curvature sets the maximum safe learning rate; the ratio of largest to smallest curvature (the condition number) sets how slowly you crawl along the flat direction. A surface with a condition number of 1000 takes roughly a thousand times longer to converge than one with a condition number of 1. Feature scaling reduces the condition number, and that is its entire practical purpose.

7 Self-check questions

Answer each question in your own words before revealing the answer.

Question 1

What is the difference between a loss function and a cost function, and which one is MSE?

Show answer
A loss function measures the error on a single training example. A cost function aggregates the losses over the whole training set, typically by averaging. MSE is a cost function: it is the average of the squared-error losses across all n examples. The per-example loss for linear regression is (yi − ŷi)²; the cost is the mean of those.

Question 2

A colleague computes the RSS of a model on a 1,000-row training set and gets 12,500. You compute the RSS of what you believe is a comparable model on a 10,000-row set and get 118,000. Which model fits better? What should you have computed instead?

Show answer
You cannot tell from RSS alone, because RSS grows with n. The correct comparison uses MSE. Your colleague's model: 12,500 / 1,000 = 12.5. Yours: 118,000 / 10,000 = 11.8. So your model actually fits slightly better per example, despite having a much larger total RSS. This is exactly the comparability problem that dividing by n solves — and it matters whenever you compare training error to validation error computed on a differently sized split.

Question 3

The MSE cost surface for linear regression is convex. State two guarantees this provides, and two things it does not guarantee.

Show answer

Guaranteed:

  • There is exactly one minimum, and it is global — no local minima to get trapped in.
  • Any point where the gradient is zero is that global minimum, so reaching a stationary point means you have solved the problem.

Not guaranteed: that gradient descent will reach the minimum quickly (ill-conditioning can make it slow), and that the resulting model will be good for your actual purpose — a convex objective can still be the wrong objective, and training error says nothing about generalisation.

Question 4

At some point on the loss surface, ∂J/∂w = +50 and ∂J/∂b = −8. In which direction should you move to reduce the cost most rapidly, and what does each sign tell you?

Show answer
The gradient is (+50, −8), which points in the direction of steepest ascent. To reduce the cost most rapidly, move in the opposite direction: decrease w and increase b, in proportion to those magnitudes — the descent direction is (−50, +8).

The signs tell you about the local slope: a positive ∂J/∂w means increasing the slope raises the cost, so the current slope is too large. A negative ∂J/∂b means decreasing the intercept raises the cost, so the current intercept is too small.

Question 5

Two features have ranges of 0–1 and 0–10,000 respectively. Sketch what the contours of the MSE surface will look like, and explain the consequences for gradient descent.

Show answer
The contours will be long, narrow, elongated ellipses aligned with the axis of the large-range feature. The cost changes extremely rapidly when you move in the direction of the large-scale coefficient and very slowly when you move in the direction of the small-scale one.

The consequences are twofold. First, the maximum safe learning rate is set by the steepest direction, so it must be small — small enough that progress along the flat direction is glacial. Second, the descent path zig-zags across the narrow valley rather than travelling along it, because the steepest direction at any point is nearly perpendicular to the direction you actually need to go. Rescaling both features to a comparable range makes the contours rounder and fixes both problems.

Question 6

The gradient at a point on the surface is exactly zero, but you are not at the minimum. Is this possible for the MSE cost function of linear regression? Is it possible in general?

Show answer
For linear regression with MSE, no. The surface is convex, so any stationary point is the global minimum. A zero gradient means you have arrived.

In general, yes. Non-convex surfaces can have saddle points — points where the gradient is zero but the surface curves upward in some directions and downward in others, like the middle of a horse's saddle. There can also be local maxima and flat plateaus. This is a central difficulty in training neural networks, and it is why convexity is worth so much when you can get it.

8 Practice exercises

Work these by hand first, then verify in the notebook.

Exercise 1 — Evaluate the cost function by hand

Use this small dataset: 

x = [1, 2, 3, 4]
y = [2, 4, 5, 7]

For each parameter pair below, compute all four predictions, all four residuals, the RSS and the MSE. Fill in the table.

w b Predictions Residuals RSS MSE
1.01.0
1.50.5
1.60.5
1.61.0

One of these four is the exact least-squares solution. Which one, and how can you tell from the residuals?

Show answers
w b Predictions Residuals RSS MSE
1.01.02, 3, 4, 50, +1, +1, +26.001.500
1.50.52.0, 3.5, 5.0, 6.50, +0.5, 0, +0.50.500.125
1.60.52.1, 3.7, 5.3, 6.9−0.1, +0.3, −0.3, +0.10.200.050
1.61.02.6, 4.2, 5.8, 7.4−0.6, −0.2, −0.8, −0.41.200.300

The exact least-squares solution is (1.6, 0.5). You can recognise it because its residuals sum to zero and the cross-products Σxiei sum to zero: (−0.1)(1) + (0.3)(2) + (−0.3)(3) + (0.1)(4) = −0.1 + 0.6 − 0.9 + 0.4 = 0. Both normal equations hold. No other row satisfies both.

Exercise 2 — Compute a gradient by hand

Using the same data as Exercise 1, compute the gradient of MSE at w = 1.0, b = 1.0.

  1. Write down the residuals.
  2. Compute Σ ei and Σ xiei.
  3. Apply the gradient formulas to get ∂J/∂w and ∂J/∂b.
  4. State the direction of steepest descent.
  5. Take a single step with α = 0.01 and confirm that the MSE decreases.
Show answers

1. Predictions at (1.0, 1.0) are 2, 3, 4, 5. Residuals: 0, +1, +1, +2.

2. Σei = 0 + 1 + 1 + 2 = 4.
Σxiei = 1(0) + 2(1) + 3(1) + 4(2) = 0 + 2 + 3 + 8 = 13.

3. ∂J/∂w = −(2/4)(13) = −6.5.
∂J/∂b = −(2/4)(4) = −2.0.
Gradient = (−6.5, −2.0).

4. Steepest descent is the negative gradient: (+6.5, +2.0) — increase both the slope and the intercept. Note that the true optimum is (1.6, 0.5), so the intercept actually needs to go down. This is a concrete illustration that the steepest direction is a local statement, not a straight line to the minimum. On an elongated surface, following the gradient exactly will overshoot and zig-zag.

5. wnew = 1.0 − 0.01(−6.5) = 1.065. bnew = 1.0 − 0.01(−2.0) = 1.02.
Predictions: 2.085, 3.150, 4.215, 5.280. Residuals: −0.085, +0.850, +0.785, +1.720. RSS = 0.0072 + 0.7225 + 0.6162 + 2.9584 = 4.3043. MSE = 1.076, down from 1.500. ✓

Exercise 3 — Read a contour plot

Figure 2 shows contours of J(w, b) for the bike-share data, with a descent path overlaid. Answer the following by reading the figure:

  1. In which general direction from the starting point does the minimum lie?
  2. Why does the path zig-zag instead of moving in a straight line?
  3. The step sizes are all the same length in parameter space. Why do later steps appear to make less progress in cost?
  4. If you rescaled the temperature feature, would the location of the minimum move? Would the shape of the contours change?
Show answers
  1. Down and to the left — that is, a smaller slope and a smaller intercept than the starting point. The start is at roughly (w ≈ 23, b ≈ −45) and the minimum at (20.65, −62.60).
  2. Because the contours are elongated. The negative gradient at any point is perpendicular to the contour passing through that point, and on a long narrow ellipse the perpendicular direction points across the valley rather than along it. Each step therefore overshoots to the other side.
  3. Because the surface flattens near the minimum. The gradient magnitude decreases as you approach the bottom, so a fixed step length covers less vertical distance in cost. The parameters are still moving by the same amount, but the cost is changing less.
  4. The minimum moves — different units give different coefficients — but the predictions do not change. The shape of the contours changes too, in the sense that the condition number changes. Which direction is "easier" to descend depends on how the scaling shifts the curvature in each parameter direction.

Exercise 4 — Design a loss surface

For each scenario below, state whether the MSE loss surface is convex, and justify your answer. Where it is convex, describe the shape of the contours.

  1. Simple linear regression with one feature, using MSE.
  2. Multiple linear regression with three features, using MSE.
  3. Multiple linear regression where two features are exact duplicates of each other.
  4. Multiple linear regression where one feature is the sum of the other two.
  5. Linear regression with a single feature and no intercept.
Show answers
  1. Convex. A quadratic in one parameter, opening upward. Contours in the two-parameter space (w, b) are ellipses.
  2. Convex. A quadratic in four parameters. Convexity is preserved under summation and composition with a linear map, so adding features does not break it. The contours live in four dimensions, which you cannot draw, but the surface still has a single minimum.
  3. Convex but not strictly convex, and the minimum is not unique. Two identical features mean you can increase one coefficient and decrease the other by the same amount without changing any prediction — so the cost is identical along that whole direction. The matrix XᵀX is singular, the normal equation has no unique solution, and the "contours" are flat-bottomed valleys rather than closed ellipses. This is perfect collinearity.
  4. Same situation. If x₃ = x₁ + x₂, the three columns are linearly dependent and there is a whole family of parameter vectors achieving the same minimum. Again convex, again not strictly convex, again no unique solution.
  5. Convex. The cost is J(w) = (1/n)Σ(yi − wxi)², a quadratic in w alone. The contours are one-dimensional, so there is nothing to plot — just a parabola with a single minimum at ŵ = Σxiyi / Σxi², exactly as derived in Tutorial 3.

Cases 3 and 4 are the ones you will meet most often in practice — usually not as exact duplicates but as near-duplicates, such as including both a "total spend" column and its two components. The surface stays convex, but it develops a long flat valley, and the fitted coefficients become unstable.

Exercise 5 — Prediction from the surface

Without Computing anything, predict the effect of each change on the shape of the MSE loss surface, and explain your reasoning.

  1. Doubling the number of training examples (same underlying distribution).
  2. Multiplying every target value by 10.
  3. Multiplying one feature by 1,000 while leaving the others alone.
  4. Removing an outlier that lies far from the current fitted line.
  5. Adding a feature that is pure random noise, uncorrelated with the target.
Show answers
  1. Same shape, same minimum location, slightly smoother. The surface is an average over examples, so adding more examples from the same distribution changes the average only by reducing sampling noise. The location of the minimum may shift a little, but the qualitative shape is unchanged.
  2. Same shape, same minimum, but the surface is stretched vertically by a factor of 100. Because J is quadratic in the target, the optimal parameters scale linearly — both w and b are multiplied by 10 — and the cost at the minimum is multiplied by 100. The contour shape in parameter space is unchanged.
  3. Same minimum location, radically different shape. The curvature in the direction of that feature's coefficient increases by a factor of a million, while the curvature in the other directions is unchanged. The contours become extremely elongated, and the condition number skyrockets. Gradient descent becomes very slow unless the learning rate is reduced — and even then it crawls along the flat directions.
  4. The minimum moves toward the remaining data, and the contours tighten. The outlier was exerting a large pull on the fit because of the squared penalty. Remove it and the least-squares solution shifts toward the bulk of the data. The curvature may also change, since the feature spread of the retained data is different.
  5. Convexity is preserved, but the minimum barely moves and the surface gains a flat direction. A noise feature has essentially no correlation with the target, so its optimal coefficient is near zero. But because it adds a parameter to the model, the surface gains a direction along which the cost changes very little — a shallow valley. This is the first hint of the problem that regularisation solves in Module 3.

9 Problem set

These problems are longer and more open-ended. They feed directly into Assignment 2 and into the optimisation work of later modules.

Problem 1 — Verify convexity algebraically and visually

This problem asks you to establish convexity from two directions.

  1. Algebraically. Expand J(w, b) in full and collect terms. Write the result in the form J = aw² + cb² + dwb + ew + fb + g, giving expressions for each coefficient in terms of the data.
  2. Show that a > 0 and c > 0, and hence that the quadratic form in (w, b) is positive semi-definite.
  3. Compute the discriminant d² − 4ac and show that it is non-positive. What does this tell you about the shape of the surface?
  4. Visually. Using the bike-share data, plot three slices through the surface: along w at the optimal b, along b at the optimal w, and along the diagonal b = w − 83. Confirm that all three are upward-opening parabolas.
  5. Explain why the third slice looks different from the first two, and why that is the source of the elongated contours.

Problem 2 — Visualise the effect of feature scaling

Using the California Housing dataset: 

from sklearn.datasets import fetch_california_housing
data = fetch_california_housing(as_frame=True)
df = data.frame

Choose two features with very different numeric ranges — for example, MedInc (median income, in tens of thousands) and Population (often in the thousands).

  1. Fit a regression on just those two features. Report the two coefficients.
  2. Build a grid over a wide range of the two coefficients and compute the MSE at each point. Plot the contours.
  3. Estimate the condition number of the surface by finding the ratio of the largest to smallest curvature. One way to do this is to compute the eigenvalues of 2XᵀX/n — the Hessian of the cost.
  4. Now standardise both features using StandardScaler and repeat. Compare the condition numbers.
  5. Run gradient descent on both surfaces, using the largest learning rate that converges in each case. Plot the cost against iteration number for both runs on the same axes.
  6. Write a paragraph explaining, in terms of the geometry of the surface, why standardising the features makes such a difference.

Problem 3 — Explore a non-convex surface

This problem is a preview of Module 5, but it uses only the tools from this tutorial.

Consider fitting a single-input model of the form ŷ = w₂x² + w₁x + b. The model is still linear in the parameters, so the MSE surface is still convex — verify this by fitting with LinearRegression on the features [x, x²] and confirming that the optimal coefficients match a direct minimisation.

Now consider instead the model ŷ = w · tanh(v x + c), where both w and v are parameters. This model is non-linear in its parameters.

  1. Generate a small synthetic dataset from a known tanh relationship plus noise.
  2. Compute the MSE over a grid of (v, w) values, holding c fixed. Plot the contours.
  3. Is the surface convex? How many local minima can you identify?
  4. Run gradient descent from five different starting points. Do they all reach the same solution?
  5. Write a short paragraph explaining what this example demonstrates about why convexity matters, and why it is not available for most modern models.

Problem 4 — The role of the number of examples

This problem explores how n affects the surface.

  1. Using the bike-share data, fit the least-squares line using only the first n points, for n = 3, 4, 5, 6, 7, 8. Record the optimal (w, b) each time.
  2. Plot the sequence of optimal points on a single contour plot of the full-data surface. What pattern do you see?
  3. For each value of n, compute the curvature of the surface at the minimum — for example, the second derivative with respect to w. How does it change as n grows?
  4. Explain the connection between your answer to part 3 and the fact that gradient descent converges faster on larger datasets (in terms of the number of iterations required, even though each iteration costs more).
  5. What happens when n = 2? Is the fit unique? What does the surface look like?

10 Discussion and projects

Discussion prompt — post 150–250 words

A convex objective can still be the wrong objective.

MSE is convex, which guarantees that any minimiser will find the best possible parameters for that criterion. But "best possible for MSE" is not the same as "best possible for your problem". Describe a prediction context in which minimising squared error would produce a model that is systematically wrong in the way that matters most — for example, a context where large errors are catastrophic rather than merely inconvenient, or where the target distribution is heavily skewed.

Then answer: what would you change about the objective, and what would you lose by changing it? Convexity, for instance, is often the first casualty.

Mini-project — build the surface explorer

Build a small interactive tool — a Jupyter notebook with ipywidgets sliders, or a simple script that regenerates plots — that lets a user explore the MSE loss surface of a simple regression.

The tool should allow the user to:

  1. Choose a dataset from at least three built-in options.
  2. Move a candidate line by adjusting w and b with sliders.
  3. See, simultaneously, the data with the candidate line, the current residuals, and the current MSE.
  4. See the position of the current (w, b) marked on a contour plot, updating live as the sliders move.
  5. Toggle between raw and standardised features, and see the contour plot change shape.

The act of building this tool will teach you more about the loss surface than any amount of reading. If you can make someone else feel the shape of the bowl, you understand it.

Discussion prompt — the cost of squaring

In Tutorial 3 you saw that squaring gives outliers disproportionate influence. That is a property of the loss function, and therefore a property of the entire surface you are minimising.

Find a real dataset with at least one genuine extreme value — not a data-entry error, but a real observation that is far from the rest. Fit a least-squares model and a least-absolute-deviations model to it. Post both sets of coefficients and describe which points the two models disagree about most. Then say which model you would defend to a stakeholder, and why.

11 Summary

What you should take away

Key terms

Loss function
A measure of error for a single training example.
Cost function
A measure of error aggregated over the whole training set, usually as an average of per-example losses. Denoted J in this course.
Objective function
Whatever quantity the optimiser is asked to minimise. Usually the cost function, but may include additional penalty terms.
Mean squared error (MSE)
The average squared residual, (1/n)Σei². The cost function for linear regression.
Loss surface
The graph of the cost function over the parameter space. For two parameters it is a three-dimensional surface; the vertical axis is cost.
Contour plot
A two-dimensional representation of a surface showing curves of equal cost. The innermost ring surrounds the minimum.
Convexity
A property of a function whose graph lies below the straight line joining any two of its points. A convex function has no local minima other than the global one.
Gradient (∇J)
The vector of partial derivatives of the cost with respect to each parameter. Points in the direction of steepest ascent.
Steepest descent
The direction −∇J, opposite to the gradient. The locally fastest way to reduce the cost.
Conditioning
How elongated the contours of a surface are. A well-conditioned surface has near-circular contours; an ill-conditioned one has long narrow ellipses.
Condition number
The ratio of the largest to smallest curvature of the surface. A high condition number means slow convergence for gradient-based methods.
Hessian
The matrix of second partial derivatives of the cost. Its eigenvalues give the curvature of the surface in each direction, and hence the largest safe learning rate.
Stationary point
A point where the gradient is zero. On a convex surface, every stationary point is the global minimum; on a non-convex surface it may be a local minimum, a local maximum or a saddle point.
Saddle point
A stationary point that curves upward in some directions and downward in others. A common obstacle in non-convex optimisation.

You are ready for Tutorial 5 when…

12 Further reading

James, Witten, Hastie & Tibshirani — An Introduction to Statistical Learning, §3.1–3.2 free online The treatment of the residual sum of squares and its minimisation. ISL does not emphasise the geometric picture, but its algebraic treatment complements this tutorial well, and its section on the Gauss–Markov theorem explains why least squares is optimal among unbiased linear estimators.
Boyd & Vandenberghe — Convex Optimization, ch. 1–3 free online The standard reference on convexity. Chapter 1 is a readable overview of what convex optimisation is and why it is special; Chapter 2 gives the mathematical definition and the operations that preserve convexity. Far more than this course requires, but the first chapter is genuinely accessible.
Deisenroth, Faisal & Ong — Mathematics for Machine Learning, ch. 7 free online Covers the calculus of optimisation, including gradients, the Hessian, Taylor series and the conditions for a minimum. Chapter 5 covers the vector calculus needed to differentiate the matrix form of the cost function. Free from the authors' website.
Goodfellow, Bengio & Courville — Deep Learning, ch. 4 free online Section 4.3 introduces the distinction between loss functions, cost functions and objectives in exactly the form used here, and Section 4.4 discusses conditioning. Written for neural networks, but the vocabulary is general.
Nocedal & Wright — Numerical Optimization, ch. 2–3 advanced The definitive treatment of gradient descent, convergence rates and conditioning. Chapter 3 gives the precise relationship between the condition number and the convergence rate that Section 3.7 of this tutorial describes informally. Beyond this course, but the source to consult if you want the full theory.
LeCun, Bottou, Orr & Müller — "Efficient BackProp" (1998) practitioner An old but remarkably durable paper on practical optimisation, including a clear explanation of why input scaling matters and what it does to the Hessian. Section 4.3 covers the conditioning argument directly and is worth reading before Tutorial 5.
scikit-learn — Preprocessing user guide reference Covers StandardScaler, MinMaxScaler and the other transformations. The "Importance of feature scaling" section gives a compact empirical demonstration of the effect on convergence.
Bishop — Pattern Recognition and Machine Learning, §3.1 textbook A more formal treatment of linear regression that includes the geometric interpretation and the maximum-likelihood justification for the squared-error criterion. Worth consulting once you are comfortable with the material in this tutorial.

What to read next

If you read one thing from this list before Tutorial 5, read Section 4.3 of the Deep Learning book. It is three pages, it introduces the loss/cost/objective distinction cleanly, and it will make the terminology used in the rest of the course unambiguous. The Boyd and Vandenberghe chapter is a rewarding follow-up if the convexity discussion left you wanting more rigour.

👤