Unit 4 gave you a model that predicts numbers. This unit is about predicting
categories — and the first thing to establish is why the tool you already have does
not work. This tutorial shows three specific ways linear regression breaks on binary
data, and introduces the reframing that fixes all three at once.
Identify a binary classification problem and explain how it differs from a regression problem.
Describe the three specific ways linear regression fails when the target is binary: unbounded output, wrong error structure and wrong loss function.
Explain why labels are not the right prediction target, and why a probability is.
Distinguish the model output (a probability) from the decision (a label produced by thresholding).
Construct a binary target from a stated problem, including the choice of which class is positive and why it matters.
Recognise the encoding convention for binary targets and state what it implies for the arithmetic.
Explain why "the model gets it right" is not a well-defined goal for a binary classifier, and what a better goal is.
Estimated time
25 minutes of reading, plus 20–25 minutes for the practice exercises and problem set. This is a conceptual tutorial — there is very little new mathematics. The value is in seeing precisely why the obvious approach fails.
2 Motivating example
A telecoms company has a churn problem. Customers leave, and the company would like to
know in advance who is likely to leave, so it can offer a retention deal before they
go. The data team has 5,000 historical customer records, each with a set of features
— tenure, monthly charge, contract type, support calls — and a single outcome:
did this customer cancel their subscription?
The outcome column contains exactly two values: 0 for stayed and
1 for cancelled. Someone on the team remembers Unit 4 and fits a linear
regression. It works. The code runs, coefficients come out, and the model produces a
number for every customer.
Then someone looks at the predictions and finds that one customer has a predicted
churn value of −0.34 and another has 1.21. What does
it mean for a customer to be −34% likely to churn? What does it mean to be
121% likely to churn?
Nothing. The model has produced numbers that cannot be interpreted, because it was
built to predict a quantity on a continuous scale and it has been pointed at a target
that can only take two values.
The target changed from a number to a label, and the model did not notice.
This is the moment in a machine learning project where the task stops being regression
and starts being classification. The rest of this tutorial is about understanding
precisely what breaks, so that the fix — introduced properly in Tutorials 2 and 3 — is
not just a new formula to memorise, but a response to a specific problem.
Regression (Unit 4)
Target is a continuous quantity
Prediction is a number on a scale
"How far off is the prediction?"
Squared error is meaningful
Predicting the mean is a sensible baseline
Classification (Unit 5)
Target is one of a finite set of categories
Prediction is a probability of belonging to a class
"Was the label assigned correctly?"
Squared error is not meaningful
Predicting the majority class is the baseline
3 Concept explanation
3.1 What makes a target binary
Unit 4 established the rule: the type of the target determines the task. A continuous
target gives regression; a categorical target gives classification. The churn outcome
is categorical with exactly two levels, so it is a binary
classification problem.
The convention throughout this unit is to encode the two classes as
0 and 1:
y ∈ {0, 1}
The class encoded as 1 is called the positive class. The choice of
which class gets the 1 is not neutral, and it is worth making deliberately.
Which class should be the positive one?
The usual convention is to make the class you are trying to detect the
positive class. In churn prediction that is "cancelled". In fraud detection it is
"fraudulent". In medical screening it is "has the condition".
This convention matters because every metric you will meet in Unit 6 — precision,
recall, sensitivity — is defined relative to the positive class. If you encode the
classes the other way round, recall means something different, and a report can
become quietly misleading. Always state which class is positive.
3.2 The arithmetic of a 0/1 target
There is one fact about a 0/1 target that does a surprising amount of work:
The mean of a 0/1 variable is the proportion of 1s.
If 620 of 5,000 customers cancelled, the mean of the churn column is
620 / 5,000 = 0.124. That number is not just an average — it is the
base rate of churn in the dataset.
This is why the mean baseline from Unit 4 becomes meaningful here. In regression, the
mean baseline predicted the average target value. For a binary target, predicting the
mean means predicting "0.124 churn" for every customer — which is the same as saying
"12.4% of customers churn" for all of them, without looking at any feature.
That is the baseline a classifier has to beat. And as you will see in Unit 6, beating
it is easy in a way that is almost embarrassing — if you are willing to accept a
metric that hides how well you are really doing.
The base rate is the number to know first
Before fitting any model, compute the base rate. If 12.4% of customers churn, then a
model that predicts "nobody churns" is 87.6% accurate. That figure will turn out to
be a trap — but it is a trap you can only see if you know the base rate in advance.
3.3 Failure one: the output is unbounded
The linear model computes ŷ = wᵀx + b,
which can be any real number. There is nothing in the arithmetic that keeps the output
within [0, 1].
For most of the data this does not matter much. If the true churn probability is
around 0.12, most predictions will land between 0 and 1 and look reasonable. The
problem appears at the edges — for customers with very long tenure, the line may
predict a negative probability; for customers with very short tenure and a high
monthly charge, it may predict a probability above 1.
Those predictions are not just imprecise. They are meaningless. There is no
interpretation of "−34% chance of churning". And once the output leaves the valid
range, every subsequent step in the pipeline — thresholding, ranking, reporting — is
working with a number that does not describe anything real.
3.4 Failure two: the error structure is wrong
This failure is subtler and more damaging than the first, and it is the reason
logistic regression exists at all.
In Unit 4, the residuals were assumed to be roughly the same size across the range of
predictions. Tutorial 6 called a violation of this heteroscedasticity,
visible as a funnel in the residual plot.
Now consider what a residual even means for a binary target. If the true label is 1
and the model predicts 0.3, the residual is 0.7. If the true label is 0 and the model
predicts 0.3, the residual is −0.3. The residual can only take two values for any
given prediction — y − ŷ is either 1 − ŷ or
−ŷ.
That means the variance of the residual is determined by the prediction, not
independent of it. Specifically, for a prediction p, the residual variance is
p(1 − p). This is largest when p = 0.5 and shrinks to zero
at the extremes.
A model that assumes constant residual variance is therefore making an assumption that
is provably false for binary data. The consequences are that the fitted coefficients
are inefficient, the standard errors are wrong, and any inference based on them is
unreliable.
Why this is the deeper problem
Failure one — unbounded output — is visible. You can look at the predictions and see
that some are negative. Failure two is invisible: the code runs, the coefficients
come out, and the model appears to work. But the error structure it assumes is
incompatible with the data, and the fit is systematically worse than it should be,
in ways that no amount of inspection of the output will reveal. This is the failure
that motivates replacing squared error with a likelihood-based loss.
3.5 Failure three: squared error is the wrong loss
Even setting aside the unbounded output and the error structure, squared error
penalises the wrong thing for a classification problem.
Consider a customer who actually churned (y = 1). Compare two models:
Model A predicts 0.9. Squared error = (1 − 0.9)² = 0.01.
Model B predicts 0.6. Squared error = (1 − 0.6)² = 0.16.
Now consider a customer who did not churn (y = 0), and the same two
models:
Model A predicts 0.9. Squared error = (0 − 0.9)² = 0.81.
Model B predicts 0.6. Squared error = (0 − 0.6)² = 0.36.
The asymmetry is instructive. A confident wrong prediction — saying 0.9 when the
answer is 0 — costs 0.81 under squared error. A confident correct prediction
— saying 0.9 when the answer is 1 — costs only 0.01. Squared error does not
distinguish strongly between "almost right" and "confidently right", and it does not
penalise "confidently wrong" as heavily as a probability model should.
The right loss for a probability is one that grows without bound as the model becomes
more confident and more wrong. Squared error is bounded: the worst it can do on any
single example is 1. A loss function that treats "predicted 0.99 for a churner" as
very different from "predicted 0.51 for a churner" would be far more appropriate.
That loss is cross-entropy, and it is the subject of Tutorial 6.
For now, the point is simply that squared error is not a natural measure of
classification error, and the model that minimises it will not be the model you want.
3.6 The reframing: predict a probability, not a label
All three failures point at the same root cause. The linear model treats the target as
a quantity to be estimated. But a binary label is not a quantity — it is the
outcome of a random event, and what you can sensibly estimate is the
probability of that event.
The reframing
Do not try to predict the label. Predict the conditional probability that
the label is 1, given the features: P(y = 1 | x).
This is a small change in wording and a large change in practice. A probability is a
number in [0, 1], so the unbounded output problem disappears once the model is
constrained to produce probabilities. A probability has a natural loss function —
likelihood — so the wrong-loss problem disappears. And a probability has an error
structure that depends on the probability itself, which is exactly what a
likelihood-based model accounts for.
In the churn example, the model's job is no longer to say "this customer will churn"
but "this customer has a 0.72 probability of churning". The first statement is a
guess about a random event; the second is an estimate of a quantity that exists even
before the event is resolved.
3.7 The model output is not the decision
Once the model produces probabilities, the question becomes: what do you do
with them?
A retention team cannot act on "72% likely to churn". It needs a yes-or-no decision:
offer the deal, or not. That decision requires a threshold. The
standard default is 0.5 — offer the deal if the predicted probability exceeds 0.5 —
but there is nothing sacred about 0.5, and in most real problems it is the wrong
choice.
Consider the churn model. A retention offer costs the company, say, £40. A customer
who leaves costs £600 in lost revenue. If the model predicts a 0.2 probability of
churn, should the team make the offer?
Expected cost of not offering: 0.2 × £600 = £120.
Cost of offering: £40.
Offering is worth it — even though the probability is well below 0.5. The right
threshold is not 0.5 but the value at which the expected cost of offering equals the
expected cost of not offering. Here that is roughly 40 / 600 ≈ 0.067.
Three separate decisions
It is worth being very clear about this, because conflating the three is the most
common conceptual error in applied classification:
The model. Estimate P(y = 1 | x) as accurately as possible.
The threshold. Choose the probability above which you will act, based on the relative costs of the two kinds of mistake.
The decision. Apply the threshold to each prediction to produce a label.
The model is a technical artefact, evaluated by how well its probabilities match
reality. The threshold is a business decision, evaluated by the consequences of the
actions it triggers. Tutorial 5 is entirely about the second, and Unit 6 is about
measuring both.
3.8 Why "the model gets it right" is not a goal
A natural first instinct is to define the goal as "assign the correct label to as many
customers as possible". This is the accuracy objective, and it is nearly always the
wrong thing to optimise.
The reason is the base rate. If 12.4% of customers churn, then a model that assigns
the label "stayed" to every single customer is 87.6% accurate. It has learned nothing
— it does not look at the features at all — and it is nonetheless correct on the vast
majority of cases. Any metric that rewards it is measuring the wrong thing.
Worse, the pressure to maximise accuracy pushes a model toward predicting the majority
class, because that is the safe choice. A model that is genuinely useful for churn
prediction will predict churn for customers whose probability is well below 0.5 — and
those predictions will mostly be wrong. The useful model will have lower
accuracy than the useless one.
A genuinely useful model might achieve: accuracy = 81%, churners correctly identified = 55%.
The second model is far more valuable to the business and scores six percentage
points lower on the metric a novice would reach for first. Unit 6 exists because
of this.
3.9 What carries over from Unit 4
It would be easy to read this tutorial as saying that Unit 4 was wasted effort. It was
not, and most of it transfers directly.
What transfers from regression to classification, and what does not
Concept
Transfers?
Notes
Framing: target, unit of observation, prediction-time features
Yes
Identical. The churn target is defined exactly the same way as a regression target.
The linear form wᵀx + b
Yes
The score is still a weighted sum of features. Logistic regression adds a link function on top.
Coefficient interpretation as partial effects
Modified
Coefficients now describe effects on log-odds rather than on the target itself. Tutorial 4 develops this.
Gradient descent as a fitting method
Yes
Logistic regression has no closed form, so gradient descent becomes the primary tool rather than the alternative.
The loss surface and convexity
Yes
The cross-entropy loss for logistic regression is also convex. Tutorial 6 makes the parallel explicit.
Train / validation / test splits, pipelines, leakage audits
Yes
Unchanged, and no less important.
Squared error as the loss
No
Replaced by cross-entropy.
RMSE, MAE, R²
No
Meaningless for a binary target. Replaced by the confusion matrix, precision, recall and ROC-AUC in Unit 6.
Residual plots
Modified
A residual can only take two values for a given prediction. Calibration curves replace them as the diagnostic of choice.
Subgroup analysis, model cards, deployment decisions
Yes
All of Tutorial 8 applies unchanged, and matters just as much here.
The workflow is the same. What changes is the loss, the metrics and the interpretation
of the coefficients. Everything else carries forward.
4 Visual illustration
Figure 1. Failure one: the output is unbounded. A straight line
fitted to binary outcomes will eventually leave the range [0, 1]. At the low end of
tenure the model predicts negative churn; at the high end it predicts probabilities
above 1. Neither prediction describes anything that exists. The shaded regions are
where the model produces values that cannot be interpreted as probabilities.
Figure 2. Failure two: the error structure is wrong. For a binary
target, the variance of the residual is p(1 − p) — a quantity
determined entirely by the prediction. It is zero when the model is certain and
largest when the model is unsure. Linear regression assumes constant variance, which
for binary data is provably false. The consequences are wrong standard errors and an
inefficient fit, and neither is visible in the output.
Figure 3. The reframing separates three decisions that are often
conflated. The linear score and the link function are technical choices, evaluated
by how well the probabilities match reality. The threshold is a business choice,
evaluated by the costs of the two kinds of error. In this example a false negative
is fifteen times as expensive as a false positive, so the optimal threshold is far
below the default 0.5.
Figure 4. The accuracy trap. On data with a 12.4% base rate, the
model that predicts "nobody churns" for every customer is 87.6% accurate and finds
zero churners. A genuinely useful model finds more than half of them and scores
six percentage points lower. Choosing a model by accuracy alone would select the
useless one. Unit 6 exists because of this figure.
5 Worked example
We will fit a linear regression to the churn data, on purpose, and watch all three
failures appear. Seeing the failures directly is more instructive than being told
about them.
Step 1 — The data and the base rate
A sample of twenty customers, with one feature and the binary churn outcome:
Twenty customer records, ordered by tenure
i
Tenure (months)
Monthly charge (£)
Churned
1
2
88
1
2
3
92
1
3
4
79
1
4
5
85
1
5
7
74
1
6
8
68
0
7
10
81
1
8
12
63
0
9
14
71
1
10
16
58
0
11
18
66
1
12
21
55
0
13
24
61
1
14
28
52
0
15
32
49
0
16
36
54
1
17
41
45
0
18
48
42
0
19
55
38
0
20
62
35
0
Number of churners (sum of the churn column)
10
Ten of the twenty customers churned, so the base rate is 0.50. This
is a deliberately balanced sample — real churn data is far more imbalanced, and we
will return to that. A balanced sample makes the arithmetic easy to follow and lets
us see the failures clearly.
Step 2 — Fit a linear regression and look at the predictions
slope : -0.01439
intercept : +0.8420
base rate : 0.500
The model is ŷ = 0.8420 − 0.01439 × tenure. A customer with zero tenure
has a predicted churn value of 0.842 — which happens to look plausible, because it
is between 0 and 1. Longer-tenure customers have lower predicted values. That is the
right direction.
Step 3 — Failure one appears at the edges
Compute the predictions at the extremes of the observed tenure range and beyond:
for tenure in [0, 2, 20, 40, 60, 70, 80]:
pred = lin.predict([[tenure]])[0]
flag = " ← outside [0,1]" if (pred < 0 or pred > 1) else ""
print(f"tenure = {tenure:>3} ŷ = {pred:>+7.4f}{flag}")
At 60 months the model predicts a churn value of −0.02. At 80 months it predicts
−0.31. These are not imprecise probabilities. They are numbers that cannot be
interpreted as probabilities at all.
Note that the failure does not require extrapolation — 60 months is well within the
range of plausible tenure values, and the dataset contains a customer with 62 months.
The line simply crosses zero on its way down, because nothing constrains it.
Step 4 — Failure two, made visible
Now look at the residuals as a function of the fitted value. Group them into three
bands by predicted value and compute the variance in each.
Residual variance by predicted value band
Predicted band
n
Mean prediction
Residual variance
0.55 – 0.85 (short tenure)
7
0.73
0.20
0.30 – 0.55 (medium tenure)
6
0.43
0.24
−0.02 – 0.30 (long tenure)
7
0.14
0.09
The variance is highest in the middle band and lowest at the extremes — which is
exactly the p(1 − p) pattern from Figure 2. At the top band, the
mean prediction is 0.73 and p(1 − p) = 0.20, which matches the
observed variance almost exactly. At the bottom band, the mean prediction is 0.14
and p(1 − p) = 0.12, which is close to the observed 0.09.
The residual variance is not constant. It is determined by the prediction. Linear
regression assumes otherwise, and that assumption is false by construction.
Step 5 — Failure three, in numbers
Compare the squared error of two models on a single customer who churned
(y = 1):
A customer who churned (y = 1)
Model A predicts 0.95 → squared error = (1 − 0.95)² = 0.0025
Model B predicts 0.55 → squared error = (1 − 0.55)² = 0.2025
Ratio: model B is penalised 81× more than model A.
A customer who stayed (y = 0)
Model A predicts 0.95 → squared error = (0 − 0.95)² = 0.9025
Model B predicts 0.55 → squared error = (0 − 0.55)² = 0.3025
Ratio: model A is penalised 3× more than model B.
The asymmetry is the problem. Being confidently correct (0.95 when the answer is 1)
is barely distinguished from being certain. Being confidently wrong (0.95 when the
answer is 0) is penalised, but only within a bounded range — the maximum possible
squared error for a single binary example is 1.
A good loss for probabilities should say that predicting 0.99 for an event that did
not occur is very different from predicting 0.55 for the same event. The
squared error difference between those two cases is 0.98 − 0.30 = 0.68, which is
substantial but not dramatic. Cross-entropy will make that difference enormous —
the subject of Tutorial 6.
Step 6 — The reframing, applied
The model above produces a score. To turn it into a probability we need a
link function that maps any real number into [0, 1] while
preserving the ordering. The standard choice is the sigmoid:
σ(z) = 1 / (1 + exp(−z))
σ(0) = 0.500
σ(+1) = 0.731
σ(+3) = 0.953
σ(−1) = 0.269
σ(−3) = 0.047
σ(±∞) → 1 or 0, but never reaches them
Applying the sigmoid to the churn scores:
Raw score, sigmoid probability, and the thresholded label
Tenure
Raw score z
p = σ(z)
Threshold 0.5
Threshold 0.3
2
+0.813
0.693
1
1
10
+0.698
0.668
1
1
20
+0.554
0.635
1
1
30
+0.410
0.601
1
1
40
+0.266
0.566
1
1
50
+0.123
0.531
1
1
58
+0.007
0.502
1
1
60
−0.021
0.495
0
1
80
−0.309
0.423
0
1
100
−0.597
0.355
0
1
120
−0.885
0.292
0
0
The output is now bounded. Every value is a legitimate probability, no matter how
extreme the input. The predictions are ordered sensibly — shorter tenure means higher
churn probability — and the model is usable.
The threshold column shows the third stage. At the default threshold of 0.5, a
customer with 60 months of tenure is classified as "will not churn". At a threshold
of 0.3, the same customer is classified as "will churn" — because a 49.5% chance of
losing £600 in revenue is worth a £40 retention offer.
Note that the sigmoid has been applied to a raw score that was itself produced by
fitting a linear regression to a binary target. That is not the correct procedure —
logistic regression fits the score and the link jointly, using maximum likelihood.
Tutorials 3 and 6 do that properly. What this table shows is the shape of
the solution, not the correct fitting method.
The same computation in code
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
df = pd.DataFrame({
"tenure": [2, 3, 4, 5, 7, 8, 10, 12, 14, 16, 18, 21,
24, 28, 32, 36, 41, 48, 55, 62],
"churned": [1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0,
1, 0, 0, 1, 0, 0, 0, 0],
})
X = df[["tenure"]].to_numpy()
y = df["churned"].to_numpy()
# --- the naive approach: linear regression on a binary target ---
lin = LinearRegression().fit(X, y)
print(f"base rate (mean of y): {y.mean():.3f}")
print(f"fitted line: ŷ = {lin.intercept_:.4f} {lin.coef_[0]:+.5f} × tenure\n")
# --- failure one: predictions outside [0, 1] ---
print("failure one — unbounded output")
for tenure in [0, 20, 40, 60, 80, 100]:
pred = lin.predict([[tenure]])[0]
out = " ← outside [0,1]" if (pred < 0 or pred > 1) else ""
print(f" tenure {tenure:>3}: ŷ = {pred:>+7.4f}{out}")
# --- failure two: residual variance depends on the prediction ---
print("\nfailure two — non-constant residual variance")
df["y_hat"] = lin.predict(X)
df["resid"] = df["churned"] - df["y_hat"]
bands = pd.cut(df["y_hat"], bins=[-1, 0.30, 0.55, 1.0],
labels=["low", "mid", "high"])
for band in ["high", "mid", "low"]:
subset = df[bands == band]
print(f" {band:>4} band: n={len(subset):>2} "
f"mean ŷ={subset['y_hat'].mean():>+.3f} "
f"residual var={subset['resid'].var():>5.3f} "
f"p(1−p)={subset['y_hat'].mean() * (1 - subset['y_hat'].mean()):>5.3f}")
# --- the reframing: score → probability → decision ---
def sigmoid(z):
return 1.0 / (1.0 + np.exp(-z))
print("\nthe reframing — score, probability, decision")
print(f"{'tenure':>7} {'score':>8} {'p = σ(z)':>10} {'τ=0.5':>7} {'τ=0.3':>7}")
for tenure in [2, 20, 40, 60, 80, 120]:
z = lin.intercept_ + lin.coef_[0] * tenure
p = sigmoid(z)
print(f"{tenure:>7} {z:>+8.3f} {p:>10.3f} {int(p > 0.5):>7} {int(p > 0.3):>7}")
The middle table is the most interesting. The observed residual variance tracks
p(1 − p) almost exactly in every band — 0.197 against 0.197,
0.243 against 0.245, 0.093 against 0.121. That is not a coincidence; it is a
mathematical identity for a binary target. The linear model is fitting data whose
error structure it cannot represent.
NBnb-09 · Why linear regression fails on binary dataFit the naive model, plot the residuals against the predictions, and see the p(1−p) pattern emerge. Then apply a sigmoid and observe the output become bounded.
6 Interactive activity
Break the model deliberately
In the Tutorial 1 notebook, you will manipulate the churn data to make each failure
more or less severe, and see how the model responds. Record what you find for each
experiment.
Extend the tenure range. Add customers with tenures of 80, 100
and 150 months, all of whom stayed. Refit and see how far below zero the line
goes. Does adding more long-tenure non-churners make the problem worse or better?
Why?
Change the base rate. Remove half the churners so the base rate
drops from 0.5 to about 0.33. Refit the linear model. What happens to the slope
and intercept? Where does the line now cross zero?
Check the accuracy of the naive model. Using a 0.5 threshold on
the naive linear predictions, compute the accuracy. Compare it against the
accuracy of always predicting "stayed" and against always predicting "churned".
Which is best, and what does that tell you about accuracy as a criterion?
Apply the sigmoid and re-threshold. Convert every naive prediction
to a probability using σ(z) and classify with a 0.5 threshold.
Does the accuracy change? Does the set of customers classified as churners change?
Why is this result interesting?
Move the threshold. With the sigmoid probabilities, compute the
classification at thresholds 0.2, 0.3, 0.5 and 0.7. For each, count how many of the
ten actual churners are correctly identified and how many non-churners are
incorrectly flagged. Tabulate the results.
Impose a realistic cost structure. Suppose a retention offer costs
£40 and a lost customer costs £600. For each of the four thresholds above, compute
the total expected cost across the twenty customers. Which threshold minimises it?
How close is it to the value predicted by the formula 40/600 ≈ 0.067?
Show the discussion
Adding long-tenure non-churners makes the problem worse. The
line must tilt downward more steeply to fit them, so it crosses zero at a lower
tenure value. The model is being pulled further out of the valid range by data
that is, in every other respect, perfectly legitimate. The failure is not caused
by bad data — it is caused by applying the wrong model to the right data.
The intercept drops and the slope flattens. With a lower base
rate, the line starts lower and the crossing point moves. The model is still
unbounded in both directions — it just fails at different points. Changing the
base rate shifts the problem without solving it.
Always predicting "stayed" wins on accuracy. At a 0.5 threshold
on the naive model, the accuracy is somewhere around 60–65% on this balanced
sample. Always predicting "churned" would be 50%. Always predicting "stayed"
would also be 50% — because the sample is balanced. On a real, imbalanced
dataset, always predicting the majority class would win outright. This is the
accuracy trap in its simplest form.
The accuracy is unchanged. The sigmoid is a monotonic
transformation, so if σ(z) > 0.5 then z > 0, and
vice versa. Every prediction above 0.5 in the sigmoid space was above zero in
the raw-score space. So the set of classified churners is identical, and the
accuracy is identical. What changed is that the outputs are now
interpretable — you can say "this customer has a 68% chance of churning"
rather than "this customer scores 0.668". The transformation alone does not
improve the model; it makes the model's output usable.
The trade-off is visible in the table. At a low threshold, more
customers are flagged, so more actual churners are caught — but more
non-churners are also incorrectly flagged. At a high threshold, the reverse. The
tabulation makes the trade-off concrete, which is the point.
The cost-minimising threshold is very low. With a 15:1 cost
ratio, the optimal threshold is around 0.067 — much lower than 0.5. At that
threshold almost every customer is flagged. That is the correct answer for the
stated costs, and it illustrates why the default threshold of 0.5 is almost
never the right one. Note also that the formula 40/600 is an approximation; the
exact optimal threshold depends on the distribution of predicted probabilities,
not just the cost ratio.
The lesson to take from the exercise
Two of the three failures — unbounded output and wrong loss — are genuine defects
in the model. The third — the threshold choice — is not a defect at all. It is a
decision the modeller must make, and making it well requires information about
costs that the data does not contain.
7 Self-check questions
Answer each question in your own words before revealing the answer.
Question 1
A dataset has a binary target, and 3% of the records are positive. A model that
predicts "negative" for every record reports 97% accuracy. Explain what is wrong
with treating this as a good result.
Show answer
The model has learned nothing. It never looks at the features, and it never
identifies a single positive case. The 97% figure simply reflects the base rate:
the positive class is rare, so predicting the majority class is almost always
correct.
Accuracy is dominated by the majority class on imbalanced data, which makes it
a poor guide to whether a classifier is useful. A model that identifies 40% of
the positives and is correct 85% of the time may be far more valuable than the
97%-accurate null model. This is the reason Unit 6 introduces precision, recall
and ROC curves: they measure the things accuracy hides.
Question 2
List the three ways linear regression fails on a binary target, and say which of
them is the hardest to detect.
Show answer
Unbounded output. The model can predict values below 0 or above 1, which cannot be interpreted as probabilities.
Wrong error structure. The residual variance is p(1 − p), which depends on the prediction, whereas least squares assumes constant variance.
Wrong loss. Squared error does not distinguish between "almost right" and "confidently right", and it is bounded, so it under-penalises confident errors.
The wrong error structure is the hardest to detect. Failure one
is visible in the predictions; failure three is a conceptual mismatch that
becomes clear once you compare loss functions. Failure two is invisible — the
code runs, the coefficients come out, the accuracy looks reasonable, and nothing
in the output reveals that the model is making an assumption that is provably
false for the data.
Question 3
Why is the mean of a 0/1 target equal to the proportion of 1s, and why does this
matter for the baseline?
Show answer
The mean is the sum divided by the count. For a 0/1 variable, the sum is exactly
the number of 1s (since the 0s contribute nothing). So the mean is
(number of 1s) / (total count), which is the proportion of 1s.
This matters because the mean baseline from Unit 4 — predict the mean for every
example — becomes "predict the base rate for every example" here. On the churn
data with a 12.4% base rate, the baseline predicts 0.124 churn probability for
every customer. That is the number any classifier must beat, and it is also the
number that makes the accuracy trap obvious: predicting the majority class
achieves 1 − 0.124 = 87.6% accuracy, which is exactly the proportion of
non-churners.
Question 4
A model outputs 0.72 for a customer. What are the two distinct things that might be
meant by this number, and why does it matter which one it is?
Show answer
The two possibilities are:
A predicted probability. The model estimates a 72% chance that
this customer will churn. This is a statement about a random event that has not
yet resolved, and it is meaningful even for a customer who, in the end, does not
churn.
A score without probabilistic meaning. The number might be a
raw output that happens to fall in [0, 1] but does not correspond to a
calibrated probability. It might order customers correctly without the values
being interpretable as likelihoods.
The distinction matters because the threshold decision depends on it. If 0.72 is
a genuine probability and the cost ratio is 15:1, the customer should obviously
receive the retention offer. If 0.72 is merely a score that happens to be high
relative to other customers, the optimal threshold is not 0.067 but some other
value determined empirically. Tutorials 2 and 3 build a model whose output
is a probability; the distinction between a score and a probability is
the subject of calibration, which is discussed in Tutorial 7.
Question 5
A colleague says: "The model predicts whether a customer will churn." What is
imprecise about that statement, and how would you restate it?
Show answer
The statement conflates two distinct things: the model's output and the decision
that follows from it. A model cannot predict a future random event; it can only
estimate the probability of that event given the available information.
A more precise restatement has three parts. The model estimates each
customer's probability of churning within the next 30 days. A retention offer is
made to any customer whose estimated probability exceeds 0.15. The 0.15 threshold
was chosen because a false negative is roughly ten times as costly as a false
positive. The first sentence describes the model; the second describes the
decision rule; the third describes why the rule is what it is. Each is a separate
choice, made by a different person for different reasons, and keeping them
separate is what makes the system auditable.
Question 6
Why does the sigmoid transformation, applied alone, not improve a model's accuracy?
Show answer
Because the sigmoid is monotonic — it never reverses the ordering
of its inputs. If one raw score is larger than another, its sigmoid is also
larger. And since it crosses 0.5 exactly at z = 0, any raw score above
zero maps to a probability above 0.5, and any raw score below zero maps to a
probability below 0.5.
The consequence is that thresholding the sigmoid output at 0.5 produces exactly
the same labels as thresholding the raw score at 0. So the accuracy, the
confusion matrix and every derived classification metric are identical.
What the sigmoid does provide is interpretability. "This customer has a
68% chance of churning" is a statement someone can act on; "this customer scores
0.668" is not. It also constrains the output to a valid range and enables a
likelihood-based loss. Those are real benefits — but they are not accuracy
improvements, and it is worth being precise about the difference.
8 Practice exercises
Work these by hand first, then verify in the notebook where computation is involved.
Exercise 1 — Identify the task
For each target below, state whether it is regression or binary classification, what
the positive class should be, and what the base rate would represent.
#
Target
Task
Positive class
Base rate means…
a
Whether a patient tests positive for a condition
b
The number of days until a machine fails
c
Whether an email is spam
d
The average rating a film will receive
e
Whether a loan applicant will default
f
Whether a passenger survived a shipwreck
g
The energy a building will consume next month
h
Whether a student will pass a module
Show answers
Target
Task
Positive class
Base rate means…
a
Binary classification
Tests positive
The prevalence of the condition in the screened population
b
Regression
—
—
c
Binary classification
Is spam
The proportion of emails in the dataset that are spam
d
Regression
—
—
e
Binary classification
Defaults
The proportion of loans that default — typically very small, often under 5%
f
Binary classification
Survived
The overall survival rate in the dataset — about 38% for the Titanic data
g
Regression
—
—
h
Binary classification
Passed
The overall pass rate in the module — often 85–95%, which means the negative class is rare
Note that for (h), the "interesting" outcome is the minority one. A model that
predicts "passes" for everyone would be 90% accurate and completely useless for
identifying students who need help. This is the same trap as the churn example,
arriving from the other direction.
Exercise 2 — The residual variance identity
For a binary target with predicted probability p, the squared residual
is either (1 − p)² or p², depending on the actual label.
Compute the expected squared residual if the label is 1 with probability p.
Simplify the expression and verify that it equals p(1 − p).
At what value of p is the residual variance maximised, and what is the maximum?
What does the variance equal when p = 0.05 and when p = 0.95?
Why does this make the assumption of constant variance in linear regression untenable for binary data?
A colleague suggests weighting each squared error by 1 / (p(1 − p))
to correct for the varying variance. What would this do? Is it a good idea?
Show answers
1. If the label is 1 with probability p, then the
squared residual is (1 − p)² with probability p, and
p² with probability (1 − p). So the expected squared residual is:
3. The function p(1 − p) is a downward-opening
parabola with its maximum at p = 0.5, where it equals
0.25.
4. At p = 0.05, variance = 0.05 × 0.95 =
0.0475. At p = 0.95, variance = 0.95 × 0.05 =
0.0475 — the same value, by symmetry. Both are about a fifth of
the maximum.
5. Linear regression's standard errors and inference assume
that the residual variance is the same for every observation. For binary data,
the variance depends entirely on the prediction, ranging from near zero when the
model is confident to 0.25 when it is uncertain. The assumption is false by
construction, so the standard errors are wrong, the coefficient estimates are
inefficient, and any inference based on them — confidence intervals, hypothesis
tests — is unreliable.
6. Weighting each squared error by 1/(p(1 − p))
would correct the heteroscedasticity. It is essentially what a weighted least
squares procedure does, and it would produce more efficient coefficient estimates.
However, it is a partial fix: the weights depend on p, which depends on
the coefficients being estimated, so the procedure becomes iterative. And it
still does not solve the bounded-output problem or provide a principled loss
function. The more elegant solution — which handles all three failures at once —
is to change the model family rather than patch the least-squares criterion.
Exercise 3 — Compute a cost-optimal threshold
A hospital uses a classifier to decide which patients to refer for a diagnostic
test. The test costs £250. A missed diagnosis costs an estimated £15,000 in later
treatment and harm to the patient.
What is the cost of a false positive?
What is the cost of a false negative?
Using the approximation τ = CFP / (CFP + CFN), compute the cost-minimising threshold.
Is this threshold higher or lower than 0.5, and why does that make sense?
Suppose the diagnostic test becomes much cheaper — £50 instead of £250. How does
the threshold change? What does that tell you about the relationship between
testing cost and the model's decision rule?
Now suppose the missed diagnosis cost rises to £50,000. What is the new threshold,
and what does it imply about how many patients are referred?
In practice, would you actually set the threshold this low? What other
considerations might raise it?
Show answers
1. A false positive means referring a patient who did not need
the test. The cost is the test itself: £250.
2. A false negative means missing a patient who did need it.
The cost is £15,000.
3.τ = 250 / (250 + 15,000) = 250 / 15,250 =
0.0164. That is a threshold of about 1.6%.
4. Far below 0.5. The cost asymmetry is extreme: a missed
diagnosis is sixty times as expensive as an unnecessary test. So the decision
rule should refer anyone whose probability of the condition is above about 1.6%
— which means referring a great many people who do not have it, in order to
catch the few who do.
5. At £50, the threshold becomes 50 / (50 + 15,000) =
0.0033, or 0.33%. Cheaper testing means a lower threshold and
more referrals. The decision rule adjusts to the economics: the cheaper it is to
check, the less certainty you need before checking.
6.τ = 250 / (250 + 50,000) = 0.005,
or 0.5%. Almost every patient with any elevated risk would be referred. At some
point the test itself becomes the bottleneck — you cannot physically test
everyone.
7. In practice the threshold would be higher, for at least three
reasons. First, the model's probabilities may not be well calibrated at such
extreme values — a predicted 1% might not really mean 1%. Second, there are
capacity constraints: if the diagnostic service can only process 200 tests a
week, the threshold must be set so that the referral volume is manageable.
Third, a false positive is not costless to the patient even if it is cheap for
the hospital — it involves anxiety, time off work and possibly invasive
follow-up. The cost formula captures the financial costs but not the human ones,
and the human costs belong in the decision too.
Exercise 4 — Score, probability, decision
For each scenario below, identify the model output, the decision that must be made,
and what information would be needed to choose the threshold.
A model estimates a 0.18 probability that a delivery will be late. The dispatcher
must decide whether to notify the customer in advance.
A model estimates a 0.83 probability that a transaction is fraudulent. The payment
system must decide whether to block it.
A model estimates a 0.44 probability that a student will fail the module. The
tutor must decide whether to schedule a check-in meeting.
A model estimates a 0.06 probability that a subscriber will cancel. The retention
team must decide whether to send a discount offer.
In each case, is the natural threshold above or below 0.5? Justify your answer from
the relative costs of the two errors, not from the probability alone.
Show answers
Notify the customer — threshold below 0.5. A false positive
means sending an unnecessary notification, which costs very little (a small
amount of customer annoyance). A false negative means the customer is
surprised by a late delivery, which damages trust. The asymmetry favours
notifying at low probabilities, so the threshold is well below 0.5 — perhaps
0.1 or even lower.
Block — threshold well above 0.5. A false positive means
blocking a legitimate transaction, which is a serious customer-service
failure and may leave someone unable to pay for something. A false negative
means a fraudulent transaction goes through, which costs the bank money but is
often recoverable. The asymmetry here is less extreme than it first appears —
both costs are substantial — but the threshold for blocking a legitimate
transaction should be high. Something like 0.8 or above.
Schedule a meeting — threshold below 0.5. A false positive
costs the tutor twenty minutes. A false negative costs a student their module
and possibly their place on the course. The asymmetry is large, so the
threshold should be low — perhaps 0.2. Note that this is exactly the trade-off
the Unit 4 tutorial on student grade prediction was about, and the same
fairness considerations apply.
Send the offer — threshold well below 0.5. A false positive
costs the discount, which is a known and bounded amount. A false negative
costs the full customer lifetime value. With the 15:1 cost ratio from the
worked example, the threshold is around 0.067. Even at a 6% predicted
probability, the expected cost of doing nothing exceeds the cost of the offer.
Notice that in three of the four cases the threshold is below 0.5, and in the
fourth it is above. There is no universal default. The right threshold is
determined by the relative costs of the two kinds of error, and those costs
are a property of the decision context, not of the model.
Exercise 5 — From regression to classification in one dataset
The following dataset records the number of study hours and the exam outcome for
twelve students.
Student
Study hours
Exam mark
Passed (mark ≥ 50)
1
3
34
0
2
5
41
0
3
6
47
0
4
8
52
1
5
10
58
1
6
11
49
0
7
13
63
1
8
15
71
1
9
16
55
1
10
18
77
1
11
20
82
1
12
22
88
1
As regression. Fit a linear model predicting exam mark from
study hours. Report the slope, intercept, RMSE and R².
As classification. Compute the base rate of "passed". What does
the mean baseline predict, and what accuracy does it achieve?
Fit a linear model predicting the pass/fail indicator from study hours. Report the
slope and intercept.
At what number of study hours does the linear model predict a pass/fail value of 0.5?
At what number of study hours does it predict a value below 0?
Which students does the linear model classify incorrectly at a 0.5 threshold?
Apply the sigmoid to the linear model's predictions. Does the set of classified
passers change?
Write two sentences describing what this exercise demonstrates about the
relationship between the two tasks. The same dataset supported both — what did
each framing reveal that the other did not?
Show answers
1. Sum of hours = 150, mean = 12.5. Sum of marks = 717,
mean = 59.75. Σ(hours × mark) = 9,734. Σ(hours²) = 2,278.
Sxy = 9,734 − 12(12.5)(59.75) = 9,734 − 8,962.5 = 771.5.
Sxx = 2,278 − 12(156.25) = 2,278 − 1,875 = 403.
Slope = 771.5 / 403 = 1.914.
Intercept = 59.75 − 1.914(12.5) = 59.75 − 23.93 = 35.82.
Model: mark = 35.82 + 1.914 × hours.
Residuals are modest; RMSE ≈ 4.3 marks, R² ≈
0.94. This is a strong fit.
2. Nine students passed, three did not. Base rate =
9/12 = 0.75. The mean baseline predicts 0.75 for every student,
which thresholds to "passed" for all twelve. That is correct for nine of them,
so accuracy = 75%. It identifies zero of the three who failed.
5. Solve 0.114 + 0.0509h = 0 →
h = −0.114 / 0.0509 = −2.24 hours. Negative study
hours is impossible, so within the range of the data the linear model never
predicts a negative value — but it would if the range were extended downwards.
This dataset does not expose failure one, because all the study hours are
positive and the intercept happens to be positive too.
6. At a threshold of 7.58 hours: students with fewer than 7.58
hours are classified as failing. Student 3 (6 hours) fails ✓. Student 6 (11
hours) passes ✓. Student 4 (8 hours) passes ✓. So the linear model classifies
correctly — all twelve. Wait — check student 6 (11 hours, actual fail). The
model predicts 0.114 + 0.0509(11) = 0.674, which is above 0.5, so it classifies
them as passing. That is incorrect. So there is one
misclassification: student 6, who studied 11 hours and
nonetheless failed. The model gets eleven of twelve correct — 92% accuracy.
7. The sigmoid is monotonic and crosses 0.5 at z = 0,
which occurs at 7.58 hours. So every student above 7.58 hours has
σ(z) > 0.5 and every student below has
σ(z) < 0.5. The set of classified passers is identical.
Accuracy is unchanged at 92%.
8. The regression framing revealed the magnitude of
the relationship — each hour of study is associated with 1.91 additional marks
— and its strength, with an R² of 0.94. That is genuinely useful for advising
students on how much study is needed to reach a target mark. The classification
framing revealed whether students fall above or below a threshold, which is what
matters if the decision is binary: pass or fail, intervene or not. The two
framings answer different questions about the same data, and neither is
universally better. The choice depends entirely on the decision the output
supports.
9 Problem set
These problems are longer and more open-ended. They feed directly into Assignment 3
and the Module 2 checkpoint.
Problem 1 — Demonstrate the three failures on real data
Choose a public binary classification dataset. Options:
Titanic (survived / not survived) — the classic beginner dataset, available from many sources.
Pima Indians Diabetes (UCI) — predicting diabetes from medical measurements.
Telco Customer Churn (Kaggle) — a larger churn dataset with realistic imbalance.
Credit Card Fraud (UCI) — an extremely imbalanced dataset, useful for showing how badly accuracy fails.
Fit a linear regression to the binary target and document each failure:
Report the base rate of the positive class.
Fit the linear model and report the coefficients.
Compute the predictions and count how many fall outside [0, 1]. What fraction is that?
Create a residual plot: residuals on the vertical axis, fitted values on the
horizontal. Describe the shape.
Bin the fitted values into four bands and compute the residual variance in each.
Compare with p(1 − p) for the mean prediction in each band.
Compute the accuracy of the linear model at a 0.5 threshold. Compare with the
accuracy of always predicting the majority class.
Apply a sigmoid to the predictions and re-threshold at 0.5. Confirm that the
accuracy is unchanged, and explain why.
Write a 200-word summary of what you found, including a sentence about which
failure was most visible and which was hardest to detect.
Problem 2 — Build a decision-cost analysis
Choose one of the datasets from Problem 1 and construct a complete cost analysis for
a realistic deployment scenario. You will need to invent plausible costs, and the
assumptions you make are part of the deliverable.
State the deployment scenario: who uses the model, and what decision does it
support? For example, a hospital screening programme, a bank's fraud team, a
retention campaign.
Estimate the cost of a false positive and the cost of a false negative. Justify
both, including what is included in "cost" — money, time, patient anxiety, customer
trust.
Compute the approximate optimal threshold using
τ = CFP / (CFP + CFN).
Apply this threshold to the model's predicted probabilities (using a proper
logistic regression for this part — or, if you have not yet covered logistic
regression, use the sigmoid-transformed linear predictions).
Compute the expected total cost at this threshold, and compare it with the
expected total cost at 0.5 and at 0.1. Tabulate the results.
Now vary the cost ratio over a range from 1:1 to 100:1, and plot the optimal
threshold against the cost ratio. What shape does the curve have? Where is it
steepest?
Write a paragraph on what the plot tells you about how sensitive the decision rule
is to the cost assumptions. If your cost estimates are off by a factor of two, how
much does the threshold move?
Finally, consider capacity constraints. If the intervention can only be delivered
to 5% of the population, what threshold would you use instead, and how does it
compare to the cost-optimal one?
Problem 3 — One dataset, two framings
Find a dataset with a continuous target that can naturally be binarised. For
example: exam marks, where you can predict the mark directly or predict pass/fail;
house prices, where you can predict the price or predict whether it exceeds some
threshold; delivery times, where you can predict the time or predict whether it will
be late.
Frame the problem as regression and fit a linear model. Report RMSE and R².
Frame the same problem as binary classification and compute the base rate.
Fit a model to the binary target. Report its accuracy, and compare it against the
majority-class baseline.
For each framing, state clearly what decision the output would support, and who
would be affected by a mistake.
Which framing is more useful for the decision you identified? Is there a decision
for which the other framing would be better?
Describe one thing the regression framing reveals that the classification framing
hides, and one thing the classification framing reveals that the regression
framing hides.
Write a short paragraph on why "which is better, regression or classification?" is
the wrong question, and what the right question is.
Problem 4 — The base rate and its consequences
This problem asks you to explore how the base rate changes everything about a
classification problem.
Take a binary classification dataset with a moderately imbalanced base rate —
somewhere between 10% and 30%.
Compute the accuracy of the majority-class baseline. How much of an advantage does
it have before any model is fitted?
Fit a linear regression to the binary target and report its accuracy. Is it better
or worse than the baseline? By how much?
Now subsample the majority class so the base rate becomes 50/50. Refit and
recompute. How does the majority-class baseline's accuracy change? How does the
fitted model's accuracy change?
Now oversample the minority class using replication so the base rate again becomes
50/50. Compare with the subsampled result. Are they the same?
Write a paragraph explaining why the base rate affects the evaluation of
a classifier even when the classifier itself is unchanged. This is a preview of
the precision–recall discussion in Unit 6.
Consider a deployment scenario where the base rate in production is different from
the base rate in training — for example, a fraud model trained on 1% fraud but
deployed in a market segment with 5% fraud. What happens to the model's precision
and recall? Would you need to retrain?
10 Discussion and projects
Discussion prompt — post 150–250 words
The threshold is not a technical parameter. It is a policy decision, and it
belongs to whoever bears the consequences of the decision.
In the churn example, the optimal threshold was determined by two numbers: the cost
of a retention offer (£40) and the value of a retained customer (£600). Those
numbers were not in the data. They came from somewhere else — from the finance team,
from the marketing budget, from a judgement about how much a customer is worth over
their lifetime.
Describe a classification decision where the threshold matters, and answer three
questions. Who chose the threshold? What information did they use? And who bears the
cost when the threshold is wrong — the organisation, the person being classified, or
someone else entirely? Then consider whether the person who bears the cost is the
same person who chose the threshold. If not, what does that imply about how the
threshold should be set, and who should be in the room when it is?
Mini-project — build a framing decision tool
Write a short interactive tool — a notebook with widgets, or a simple command-line
script — that guides a user through the decision of whether a given problem is
regression or classification, and if classification, what the threshold should be.
The tool should ask:
What is the quantity you want to predict?
Is it a continuous quantity, or one of a finite set of categories?
If categorical, how many categories, and are they ordered?
What decision will be made with the output?
What happens if the model is wrong in each direction — too high and too low, or false positive and false negative?
Estimate the cost of each error type.
Compute and display the cost-optimal threshold, with a warning about the assumptions.
The value of building this tool is that it forces you to specify, in code, the
questions you should be asking. A practitioner who can run through this checklist in
their head — without the tool — is a practitioner who will not accidentally fit a
linear regression to a binary target.
Discussion prompt — the limits of a probability
A model says a customer has a 0.72 probability of churning. What does that actually
mean for this specific customer?
There are two readings, and they are both defensible. One is frequentist:
among all customers like this one — same tenure, same charge, same contract — about
72% will churn. The other is subjective: the model's degree of belief that
this customer will churn is 0.72, given everything it knows. Post your view on which
reading is more useful for the retention team, and respond to a classmate who takes
the other position. Neither answer is wrong, but the two readings lead to different
conclusions about what the model can be used for. A frequentist probability is about
a group; a subjective one is about an individual. The retention team acts on
individuals.
11 Summary
What you should take away
A target with two values is a binary classification problem, not
a regression problem. The two tasks require different models, different losses and
different metrics.
Encode the two classes as 0 and 1, and state which is the
positive class — because every classification metric is defined
relative to it.
Linear regression fails on binary targets in three ways:
unbounded output, wrong error structure, and the wrong loss function.
The error structure failure is the subtlest: the residual
variance for a binary target is p(1 − p), determined by the
prediction, whereas least squares assumes constant variance. This is provably
false and invisible in the output.
The reframing is to predict the conditional probability
P(y = 1 | x) rather than the label. A probability is
bounded, has a natural loss function, and has an error structure that a
likelihood-based model accounts for.
The model output and the decision are separate. The output is a
probability; the decision is a label produced by thresholding. The threshold is a
business choice determined by the relative costs of the two kinds of error, not by
a mathematical rule.
Accuracy is a trap on imbalanced data. A model that predicts the
majority class for everyone can be very accurate and completely useless. Unit 6
exists because of this.
Almost everything from Unit 4 carries forward: framing, the linear form, gradient
descent, the loss surface, splits, pipelines, subgroup analysis, model cards and
deployment decisions. What changes is the loss, the metrics and the interpretation
of the coefficients.
The three failures and their fixes
Why linear regression fails, and what replaces it
Failure
Symptom
Why it happens
Fix
Unbounded output
Predictions below 0 or above 1
The linear form has no constraint on its range
Apply a link function (sigmoid) that maps the score into [0, 1]
Wrong error structure
Residual variance tracks the prediction, following p(1 − p)
For a binary target, the variance is determined by the mean
Use a likelihood-based loss that models the variance as a function of the prediction
Wrong loss
Squared error is bounded, so it under-penalises confident errors
Squared error was designed for continuous targets
Use cross-entropy, which grows without bound as confidence and error diverge
Key terms
Binary classification
A supervised task in which the target has exactly two possible values, conventionally encoded as 0 and 1.
Positive class
The class encoded as 1. By convention, the class you are trying to detect. Every classification metric is defined relative to it.
Base rate
The proportion of positive cases in the dataset. Equal to the mean of a 0/1 target.
Majority-class baseline
A classifier that predicts the more common class for every example. Its accuracy is 1 − base rate, which makes it a deceptively strong baseline on imbalanced data.
Conditional probability P(y = 1 | x)
The probability that the label is 1 given the feature vector. The quantity a classifier should estimate.
Link function
A function that maps the unbounded linear score onto the range of valid probabilities. The sigmoid is the standard choice for binary classification.
Sigmoid function
σ(z) = 1 / (1 + e−z). A monotonic function mapping any real number to a value strictly between 0 and 1.
Threshold (τ)
The probability above which the classifier predicts the positive class. A decision parameter, not a property of the model.
Heteroscedasticity
Non-constant variance of the residuals. For a binary target this is inherent: the variance is p(1 − p).
Accuracy trap
The tendency for accuracy to reward models that predict the majority class on imbalanced data, making it a misleading metric for classification.
Calibration
The agreement between a model's predicted probabilities and the observed frequencies. A model that predicts 0.7 for 100 cases should see roughly 70 positives among them.
Decision rule
The rule that converts a probability into an action. Comprises a threshold and the action it triggers. Separate from the model.
You are ready for Tutorial 2 when…
You can identify a binary classification problem and state the positive class.
You can list the three ways linear regression fails on binary data and explain why each occurs.
You can explain why the residual variance equals p(1 − p) for a binary target.
You can compute the base rate and the majority-class accuracy for a dataset.
You can explain why the model output and the decision are separate, and how the threshold is chosen.
You can compute a cost-optimal threshold from the relative costs of the two error types.
You can explain the accuracy trap to someone who has not seen it before.
12 Further reading
James, Witten, Hastie & Tibshirani — An Introduction to Statistical Learning, ch. 4 free online
Section 4.1 and 4.2 motivate logistic regression with the same argument developed
here — the bounded-output and error-structure problems — and then introduce the
sigmoid and maximum likelihood. Read it after this tutorial to see the same ideas
presented with slightly more mathematical detail.
Hastie, Tibshirani & Friedman — The Elements of Statistical Learning, §4.4 free online
A more rigorous treatment of the logistic regression setup, including the
log-odds formulation and the justification for maximum likelihood. Section 4.4.1
on the logit transformation is the natural companion to Tutorial 4 of this unit.
Murphy — Probabilistic Machine Learning: An Introduction, ch. 10 free online
Chapter 10 covers logistic regression from a probabilistic perspective, including
the Bayesian view and the relationship between the sigmoid and the exponential
family. More advanced than this course requires, but the first half of the chapter
is a good complement to the material here.
King & Zeng — "Logistic Regression in Rare Events Data" (2001) advanced
The standard reference on what happens to logistic regression when the positive
class is rare — which is the situation in fraud detection, medical screening and
many other real classification problems. Dense, but the introduction and the
figures make the core point clearly.
Provost & Fawcett — Data Science for Business, ch. 7–8 textbook
A practitioner's account of classification that emphasises the decision and the
cost structure rather than the algorithm. Chapter 7 on evaluating classifiers is
particularly good on the accuracy trap and the reason precision and recall exist.
Written for a business audience, which makes the framing material unusually clear.
scikit-learn — Linear Models user guide, logistic regression section reference
The documentation for LogisticRegression, including the note that
scikit-learn regularises by default — a detail that surprises many users. The
examples showing the effect of the regularisation parameter on a binary dataset
are a useful preview of Unit 6.
Niculescu-Mizil & Caruana — "Predicting Good Probabilities with Supervised Learning" (2005) practitioner
An empirical study comparing the calibration of different classifiers. The finding
that logistic regression is unusually well calibrated — better than many more
complex models — is worth knowing, and it is a strong argument for using logistic
regression as the baseline classifier in any project.
Gelman, Hill & Vehtari — Regression and Other Stories, ch. 13 textbook
A modern, readable treatment of logistic regression that emphasises interpretation
over computation. The discussion of how to present logistic regression results to
a non-technical audience is directly useful, and the chapter is short.
What to read next
If you read one thing from this list before Tutorial 2, read Section 4.1 of
An Introduction to Statistical Learning. It covers the same ground as this
tutorial in about eight pages, with slightly more mathematical detail, and it will
make the sigmoid and maximum-likelihood material that follows more comfortable. The
Provost and Fawcett chapter is the best complement if you want more on the decision
framing, which is the part of this material that most practitioners get wrong.
The technical content of Unit 4 ends here. What carries forward into Unit 5 and
beyond is not the algebra — it is the workflow, the diagnostics, and the habit of
asking what a model does to the people it is applied to.