Decision Boundaries

01 Learning objectives

02 Motivating example

A hospital is piloting a screening model for a condition that benefits from early detection. Two biomarkers are cheap to measure, and a logistic regression has been fitted to historical patient records:

z = −8 + 0.3·x₁ + 0.5·x₂

where x₁ is the concentration of biomarker A (measured on a scale from 0 to 25) and x₂ is the concentration of biomarker B (scale 0 to 20). The model outputs the probability that the patient has the condition.

The clinicians like the probability. It lets them triage — a patient at 0.95 goes straight to confirmatory testing, a patient at 0.05 is reassured and sent home. But at some point the hospital needs an automatic rule. The screening programme runs at scale; not every patient can have a conversation with a specialist before a decision is made.

So the question becomes: where do we draw the line?

The obvious answer is 0.5 — predict positive if the probability is at least half. But this raises a geometric question that turns out to be far more informative than it first appears. In the space of the two biomarkers, the set of patients for whom the model predicts exactly 0.5 is a specific curve. Which curve? What shape does it have? What does its shape tell us about the model?

And a second question follows immediately. If the clinicians decide that 0.5 is too conservative — that they would rather catch more cases at the cost of more false alarms — how exactly does the curve change? Does it tilt? Does it move? Does it become a different shape altogether?

The answers to these questions are what this tutorial is about. And they matter far beyond this one hospital: the shape of the decision boundary is the clearest way to see what any classifier has actually learned.

03 Concept explanation

3.1 From probability to decision

A logistic regression outputs p = σ(z), a number between 0 and 1. To produce a class label, we apply a threshold t:

• Predict the positive class if p ≥ t.
• Predict the negative class if p < t.

The conventional default is t = 0.5, and in the absence of any other information it is a reasonable starting point — it treats the two classes symmetrically. But it is not sacred. In T-06 you will see contexts where 0.5 is clearly wrong: fraud detection, cancer screening and spam filtering all call for different thresholds because the costs of the two kinds of error are not equal.

For now, keep the threshold general. Everything in this section works for any t.

3.2 The boundary equation

The decision boundary is the set of feature values at which the model is exactly indifferent — the points where the predicted probability equals the threshold.

p = t

Applying the logit to both sides — valid because the logit is a strictly increasing function — gives:

z = logit(t) = log( t / (1 − t) )

Substituting the definition of z:

β₀ + β₁x₁ + β₂x₂ + … + βₚxₚ = logit(t)

This is the equation of the decision boundary. Rearranged:

β₀ + β₁x₁ + β₂x₂ + … + βₚxₚ − logit(t) = 0

The left-hand side is a linear function of the features. The right-hand side is a constant. So the boundary is defined by a linear equation — which is why logistic regression is called a linear classifier, despite its non-linear probability output.

β₀ + β₁x₁ + … + βₚxₚ = 0

3.3 Geometry: what shape is the boundary?

The shape of the boundary depends only on how many features the model uses. The table below summarises the possibilities.

The word hyperplane is the general term: a flat surface of one dimension less than the space it lives in. In one dimension it is a point, in two a line, in three a plane, and in higher dimensions the generalisation you cannot visualise but can still work with algebraically.

Whatever its dimension, a hyperplane divides the space into exactly two decision regions — the positive region on one side, the negative region on the other. The model predicts the positive class everywhere in the first region and the negative class everywhere in the second.

3.4 What the sigmoid does and does not control

This is the central insight of the tutorial, so it is worth stating separately.

To see why, imagine replacing the sigmoid with any other strictly increasing function g that satisfies g(0) = 0.5 — the probit function, for example, or a simple linear ramp clipped to [0, 1]. The model still predicts positive whenever g(z) ≥ 0.5. Because g is strictly increasing and g(0) = 0.5, this happens exactly when z ≥ 0. The boundary is z = 0 in every case.

What the link function does control is how the probability changes as you move away from the boundary. Near the boundary, the sigmoid rises steeply; far from it, it flattens. A different link function would produce a different probability field while leaving the boundary unchanged.

The practical implication: when you look at a plot of a logistic regression's decision boundary, you are seeing a property of the coefficients, not of the sigmoid. The sigmoid is invisible in that plot.

3.5 Moving the threshold

What happens to the boundary when the threshold changes? Substituting t = 0.9 into the boundary equation:

β₀ + β₁x₁ + … + βₚxₚ = logit(0.9) = log(9) ≈ 2.197

compared with the default threshold t = 0.5:

β₀ + β₁x₁ + … + βₚxₚ = logit(0.5) = 0

The two equations differ only in the constant on the right-hand side. In geometric terms, the boundary has shifted parallel to itself — same orientation, same slope, different position. Raising the threshold moves the boundary towards the negative region (the model becomes more reluctant to predict positive); lowering it moves the boundary towards the positive region.

Three things do not change when the threshold changes:

• The coefficients. The model is identical. Only the decision rule has changed.
• The probability assigned to each case. Every point in feature space still receives the same predicted probability it did before.
• The ranking of cases. If one case has a higher predicted probability than another before the threshold change, it still does afterwards.

What changes is only the binary label assigned to each case. This is a crucial distinction for T-06, where you will see that threshold tuning is a decision-making activity, not a modelling one.

3.6 Non-linear boundaries through feature engineering

So far the boundary has always been a hyperplane. But logistic regression can produce curved boundaries — without changing the model form at all. The trick is to add derived features.

Suppose we add the squares and the product of the two features:

z = β₀ + β₁x₁ + β₂x₂ + β₃x₁² + β₄x₂² + β₅x₁x₂

The model is still linear in its parameters — each βⱼ appears to the first power — and it can still be fitted by the same machinery as before. But the boundary is now:

β₀ + β₁x₁ + β₂x₂ + β₃x₁² + β₄x₂² + β₅x₁x₂ = 0

which is the general equation of a conic section. Depending on the coefficients, this can be a circle, an ellipse, a parabola or a hyperbola. The decision regions are no longer half-planes but the inside and outside of a curve.

This is the standard way logistic regression handles problems that are not linearly separable. You are not changing the model family; you are changing the space in which the model operates. T-03's sibling topic in Module 3 — support vector machines — uses the same idea, expressed through kernels rather than explicit features.

3.7 A limitation worth knowing: XOR

Not every problem can be solved this way, and the classic counterexample is worth understanding because it reappears in Module 5.

The XOR problem has four cases and two binary features:

Plot the four points: the two negatives sit at opposite corners of a square, and the two positives sit at the other two corners. No straight line can put both positives on one side and both negatives on the other. A logistic regression on (x₁, x₂) alone simply cannot represent this function, however long you train it.

But add the product feature x₁x₂ and the problem becomes trivial. Consider z = x₁ + x₂ − 2x₁x₂ − 0.5:

• (0,0): z = −0.5, negative ✓
• (0,1): z = 0.5, positive ✓
• (1,0): z = 0.5, positive ✓
• (1,1): z = −0.5, negative ✓

The lesson generalises. Logistic regression is a linear classifier in the space you give it. If you can supply the right features, it can express highly complex boundaries. If you cannot, it is fundamentally limited — a constraint that motivated the layered architectures you will meet in Module 5.

04 Visual illustration

Figure 1 shows the screening model from Section 02 in the space of its two biomarkers. The straight line is the decision boundary at the default threshold. The shading marks the region where the model predicts the positive class.

Reading the plot

Four things are worth noting.

The boundary is straight. With two features and no derived terms, the boundary is a line. Points on the line receive p = 0.5 exactly.

The boundary passes through the middle of the overlap. It is not a perfect separator — some positive cases sit just inside the negative region, and vice versa. That is expected with real data.

Distance from the boundary maps to confidence, but not linearly. A point just inside the positive region might have p = 0.55; a point far into it might have p = 0.99. The probability rises quickly near the boundary and then saturates — the sigmoid behaviour from T-02.

The boundary is a property of the coefficients, not the data. The points show where the data sits; the line shows what the model has learned. A different model fitted to the same data would draw a different line.

Moving the threshold

Figure 2 shows the same model with three different thresholds. As the threshold rises, the boundary shifts towards the negative region; as it falls, the boundary shifts towards the positive region. In every case the lines remain parallel.

The key observation is that Figure 2 contains exactly one model. The three lines are three different rules applied to the same probabilities. This is why threshold selection belongs to the deployment conversation, not the modelling one.

05 Worked example

z = −8 + 0.3·x₁ + 0.5·x₂

−8 + 0.3x₁ + 0.5x₂ = 0

0.5x₂ = 8 − 0.3x₁
x₂ = 16 − 0.6x₁

z = −8 + 0.3(12) + 0.5(12)
  = −8 + 3.6 + 6.0
  = 1.6

p = 1 / (1 + e^(−1.6))
  = 1 / (1 + 0.2019)
  = 1 / 1.2019
  = 0.832

logit(0.2) = log( 0.2 / 0.8 )
           = log(0.25)
           ≈ −1.386

−8 + 0.3x₁ + 0.5x₂ = −1.386

0.3x₁ + 0.5x₂ = 6.614

x₂ = 13.228 − 0.6x₁

06 Interactive activity

import numpy as np
import matplotlib.pyplot as plt

x1 = np.linspace(0, 25, 200)

def boundary(b0, b1, b2, x1):
    # b0 + b1*x1 + b2*x2 = 0  ->  x2 = -(b0 + b1*x1) / b2
    return -(b0 + b1 * x1) / b2

fig, ax = plt.subplots(figsize=(7.5, 5))

configs = [
    (-8, 0.3, 0.5, "baseline"),
    (-8, 0.6, 0.5, "beta1 doubled"),
    (-8, 0.3, 1.0, "beta2 doubled"),
    (-4, 0.3, 0.5, "intercept halved"),
]

for b0, b1, b2, label in configs:
    ax.plot(x1, boundary(b0, b1, b2, x1), linewidth=2.4, label=label)

ax.set_xlim(0, 25)
ax.set_ylim(0, 20)
ax.set_xlabel("biomarker A  (x1)")
ax.set_ylabel("biomarker B  (x2)")
ax.set_title("Decision boundaries for varying coefficients")
ax.legend()
ax.grid(alpha=0.25)
plt.tight_layout()
plt.show()

The answers, which you should confirm against your plots:

• Doubling β₁ makes the slope more negative. The line rotates downwards — biomarker A becomes a stronger substitute for biomarker B.
• Doubling β₂ makes the slope less negative. The line rotates the other way — biomarker B becomes more influential relative to biomarker A.
• Halving the magnitude of β₀ shifts the line upward. The model becomes more willing to predict positive at low feature values.
• β₁ = 0 gives a horizontal line at x₂ = −β₀/β₂. The model ignores biomarker A entirely — only B matters.
• β₂ = 0 gives a vertical line at x₁ = −β₀/β₁. Only biomarker A matters. Note that this case is special: setting β₂ = 0 removes x₂ from the model, so the "line" no longer depends on x₂ at all.

The general rule: coefficients control the orientation of the boundary; the intercept and threshold control its position. More precisely, the vector (β₁, β₂, …, βₚ) is perpendicular to the boundary, so changing a coefficient tilts it, while changing β₀ or t slides it.

07 Self-check questions

Answer each question before revealing the solution. If you get one wrong, re-read the relevant subsection rather than simply moving on.

08 Practice exercises

Work these by hand first, then check your answers computationally.

1. A model has z = −4 + 0.5x₁ + 0.25x₂. Write the boundary equation at t = 0.5 in slope-intercept form (x₂ as a function of x₁).
2. For the same model, classify the points (5, 7), (2, 5) and (8, 3), and compute the predicted probability for each.
3. Using the model from exercise 1, find where the boundary crosses the x₁-axis and where it crosses the x₂-axis.
4. A model has z = −6 + x₁ + x₂. Find the boundary equation at t = 0.9.
5. For the model z = −5 + 0.4x₁ + 0.5x₂, what threshold would place the point (5, 3) exactly on the boundary?
6. A logistic regression has β₁ = 0. Is the decision boundary vertical, horizontal, or neither? Explain your reasoning, then answer the same question for β₂ = 0.
7. A model has z = −2 + 0.1x₁ + 0.1x₂ + 0.02x₁² + 0.02x₂². Is the boundary linear? What shape is it likely to be, and why?
8. A model's boundary at t = 0.5 is x₂ = 10 − 2x₁. Sketch the positive and negative regions and mark one point on each side that is far from the boundary. What are the approximate probabilities at those two points, given that the model's score at a distance d perpendicular to the boundary grows proportionally to d?

09 Problem sets

These problems require reasoning about model behaviour rather than computation. Several have more than one defensible answer; what matters is the quality of the argument.

10 Discussion and projects

Discussion questions

1. Is the decision boundary a property of the model or of the decision-maker? The coefficients belong to the model; the threshold belongs to whoever deploys it. Where does the boundary itself belong?
2. Logistic regression produces a linear boundary. Is this best understood as a limitation or as a feature? What does it buy you that a more flexible model would not?
3. A point far from the boundary has a predicted probability close to 0 or 1. Does this mean the model is more likely to be correct about that point? Discuss, considering the possibility that the model itself is wrong.
4. In medical screening, would you rather move the boundary towards the positive region or the negative region? Does the answer depend on the disease, the treatment, and the patient? Construct two contrasting cases.

Mini-project · Visualising decision boundaries in scikit-learn

In this project you will fit a logistic regression to a two-dimensional dataset and plot both the decision boundary and the surrounding probability field. The visual is one of the most useful diagnostic tools in applied machine learning, and it directly connects the algebra of this tutorial to the geometry of a real dataset.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.preprocessing import PolynomialFeatures

X, y = make_classification(n_samples=250, n_features=2, n_redundant=0,
                           n_informative=2, n_clusters_per_class=1,
                           class_sep=1.1, random_state=0)

clf = LogisticRegression().fit(X, y)

xx, yy = np.meshgrid(
    np.linspace(X[:, 0].min() - 0.8, X[:, 0].max() + 0.8, 300),
    np.linspace(X[:, 1].min() - 0.8, X[:, 1].max() + 0.8, 300),
)
grid = np.c_[xx.ravel(), yy.ravel()]
Z = clf.predict_proba(grid)[:, 1].reshape(xx.shape)

fig, ax = plt.subplots(figsize=(7.5, 5.5))
contour = ax.contourf(xx, yy, Z, levels=20, cmap="Blues", alpha=0.75)
ax.contour(xx, yy, Z, levels=[0.5], colors="#1f4e8c", linewidths=2.6)
ax.scatter(X[y == 0, 0], X[y == 0, 1], marker="o", facecolors="none",
           edgecolors="#5b6472", s=45, linewidths=1.6, label="negative")
ax.scatter(X[y == 1, 0], X[y == 1, 1], marker="s", c="#0f7b6c",
           s=42, label="positive")
ax.set_xlabel("feature 1")
ax.set_ylabel("feature 2")
ax.legend(loc="upper left")
plt.colorbar(contour, ax=ax, label="predicted probability")
plt.tight_layout()
plt.show()

As you work through the project, keep four questions in mind. Is the boundary a straight line, and why must it be? What do the contour colours actually represent? Are there misclassified points, and what are their predicted probabilities? And when you add polynomial features, does the boundary remain linear — and if it becomes curved, has the model become non-linear?

11 Summary

This tutorial turned the probability model of T-02 into a classifier, and examined the geometry that results.

• A classifier is a probability plus a threshold. Predict positive when p ≥ t. The default is 0.5, but the choice of threshold is a decision-making problem, not a modelling one.
• The decision boundary is where p = t. Equivalently, β₀ + β₁x₁ + … + βₚxₚ = logit(t). At the default threshold this reduces to the linear score being zero.
• The boundary is a hyperplane. A point in one dimension, a line in two, a plane in three, and the generalisation in higher dimensions. It divides the feature space into exactly two decision regions.
• The sigmoid does not affect the boundary. Any strictly increasing link function with g(0) = 0.5 produces the same boundary at the default threshold. The link function shapes the probability field, not the decision surface.
• Changing the threshold moves the boundary parallel. The coefficients, the probabilities and the ranking of cases are all unchanged; only the labels change.
• Changing a coefficient rotates or shifts the boundary differently. The coefficient vector is perpendicular to the boundary, so altering it tilts the surface rather than sliding it.
• Non-linear features produce non-linear boundaries. Adding squares or products of the original features makes the boundary a conic section, while the model remains linear in its parameters.
• Not every problem is linearly separable. XOR cannot be solved by logistic regression on its raw features. This limitation is the starting point for the layered architectures in Module 5.

12 Further reading

Core

• James, Witten, Hastie & Tibshirani — An Introduction to Statistical Learning, Chapter 4. Section 4.3 covers logistic regression, and Section 4.4 on linear discriminant analysis provides a useful contrast: a different model form that produces the same family of linear boundaries.
• Hastie, Tibshirani & Friedman — The Elements of Statistical Learning, Section 4.4. A more rigorous treatment, including the connection to generalised linear models and to the geometry of separating hyperplanes.
• scikit-learn documentation — LogisticRegression, and the example gallery entries on decision boundaries. The gallery contains several worked examples plotting decision boundaries for different classifiers on the same data — a useful side-by-side comparison.

Supplementary

• Bishop — Pattern Recognition and Machine Learning, Chapter 4. A probabilistic treatment that develops the geometry of linear decision boundaries in detail and connects them to the broader family of linear models for classification.
• Duda, Hart & Stork — Pattern Classification, Chapters 4–5. The classic treatment of linear discriminants and the geometry of decision surfaces, with extensive discussion of the XOR problem and its significance.
• Provost & Fawcett — Data Science for Business, Chapter 7. An accessible discussion of threshold selection as a business decision, with concrete examples from fraud detection and direct marketing.

On the history and limits of linear separability

• Minsky & Papert — Perceptrons (1969). The book that famously demonstrated the limits of single-layer linear classifiers, using XOR as the central example. Its influence — and the subsequent revival of neural networks — is essential context for Module 5.
• Cover, T.M. — "Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition" (1965). The classic paper on the capacity of linear classifiers in high-dimensional spaces, and why the XOR limitation fades as the number of features grows.