Probability and the Sigmoid Function

01 Learning objectives

02 Motivating example

A bank is building a model to decide whether to approve loan applications. For each applicant the bank holds a credit score, a debt-to-income ratio, and a handful of other variables. The decision is binary — approve or decline — but the bank is not really interested in a binary answer. It is interested in a probability.

Why? Because a probability can be used for far more than a yes/no decision:

• Pricing. An applicant with a 6% probability of default might receive a different interest rate than one with a 2% probability, even though both are approved.
• Threshold setting. The bank can choose where to draw the line between approve and decline, and can move that line depending on its appetite for risk — a decision that is impossible if the model only ever emits True or False.
• Ranking and review. Applications near the threshold can be sent to a human underwriter, while those far from it are auto-processed.
• Regulatory explanation. "Your application was declined because the estimated probability of default was 0.71" is a far more defensible statement than "the model said no."

Suppose a colleague suggests the obvious approach: fit a linear regression to the binary outcome, where default is coded as 1 and repayment as 0. The predicted value is then interpreted as a probability. On the surface this seems reasonable — the model is linear, the coefficients are interpretable, and the machinery is already familiar from Unit 4.

It does not work. The next section explains precisely why, and what replaces it.

03 Concept explanation

3.1 Why linear regression fails at classification

Three problems, in increasing order of severity.

The range problem. A linear function z = β₀ + β₁x₁ + … + βₚxₚ takes values on the whole real line. A probability must lie in [0, 1]. Nothing in the linear model constrains its output, so any prediction outside the unit interval is meaningless.

The shape problem. Even when a linear fit happens to stay inside [0, 1] for the observed data, its behaviour at the edges is wrong. Probabilities do not change at a constant rate: they saturate. An applicant with a credit score of 620 and one with 640 differ meaningfully in risk; an applicant with 800 and one with 820 barely differ at all, because both are already near-certain to repay. A straight line cannot express this saturation.

The statistical problem. Linear regression assumes errors are normally distributed with constant variance. For a binary outcome, the error can only take two values at each point, and its variance depends on the predicted probability — it is largest when p = 0.5 and shrinks to zero as p approaches 0 or 1. The assumptions are simply violated, which means the coefficients are estimated inefficiently and the standard errors are wrong.

3.2 The idea: keep the linear part, transform the output

The linear model is not useless — it is just being asked to do the wrong job. What it is genuinely good at is producing a score: a single number that summarises the evidence, higher meaning "more likely to be positive." What we need is a way to turn that score into a probability.

So we split the model into two stages:

Stage 1 — linear score:      z = β₀ + β₁x₁ + β₂x₂ + … + βₚxₚ
Stage 2 — squash to [0,1]:   p = σ(z)

The function σ is the sigmoid, also called the logistic function. The full model is p = σ(β₀ + β₁x₁ + … + βₚxₚ), and it is called — unsurprisingly — logistic regression. The name is a historical accident: it is a classification model, despite containing the word "regression."

3.3 The sigmoid function

The sigmoid is defined as:

σ(z) = 1 / (1 + e^(−z))

Its properties are exactly what classification requires:

The sigmoid has the familiar S-shape: nearly flat at both ends, steepest in the middle. That flatness at the extremes is precisely the saturation behaviour that linear regression could not produce.

3.4 The logit — running the sigmoid backwards

Because the sigmoid is strictly increasing, it has an inverse. Rearranging p = 1 / (1 + e^(−z)) gives:

z = log( p / (1 − p) )

This quantity has a name. The ratio p / (1 − p) is the odds, and its logarithm is the log-odds or logit.

This is the key insight of the whole tutorial: logistic regression is a linear model in the log-odds. The model says that

log( p / (1 − p) ) = β₀ + β₁x₁ + … + βₚxₚ

which is a genuinely linear statement — just not on the scale you might have expected. The linearity lives on the log-odds scale, and the sigmoid is the bridge back to probabilities.

3.5 Probability, odds and log-odds

These three ways of expressing the same thing appear constantly in logistic regression, and mixing them up is the most common source of confusion. The table below sets them side by side.

Notice the pattern in the ranges. Probability is bounded above and below; odds are bounded below but not above; log-odds are unbounded in both directions. Only the log-odds scale can be modelled by an unrestricted linear function, which is exactly why the model is expressed there.

3.6 Interpreting the coefficients

Because the model is linear in log-odds, a coefficient βⱼ has a clean interpretation:

• On the log-odds scale: a one-unit increase in xⱼ, holding all other features constant, increases the log-odds by βⱼ.
• On the odds scale: the odds are multiplied by e^(βⱼ) — the odds ratio.
• Sign: βⱼ > 0 means the feature increases the probability of the positive class; βⱼ < 0 means it decreases it; βⱼ = 0 means no effect.

The odds-ratio interpretation is worth internalising. A coefficient of 0.693 corresponds to an odds ratio of e^0.693 ≈ 2 — the odds double for each one-unit increase. A coefficient of −1.386 corresponds to e^(−1.386) ≈ 0.25 — the odds fall to a quarter.

04 Visual illustration

The figure below plots the sigmoid function. Read it carefully — almost everything in the rest of this unit can be understood by referring back to it.

Reading the curve

Three features of the plot are worth pointing out explicitly.

The middle is steep. Around z = 0, a small change in the score produces a relatively large change in probability. This is the region of genuine uncertainty, where additional evidence is most informative.

The ends are flat. By z = −4 the curve is already almost touching zero; by z = 4 it is almost touching one. Additional evidence in these regions changes the probability very little — the model is already confident.

The output never reaches the limits. However large z becomes, σ(z) never equals exactly 0 or exactly 1. This is a feature, not a limitation: it means the model never claims absolute certainty, and it keeps the mathematics well-behaved (in particular, the log-odds never become infinite).

The table below gives numerical values for the curve, which are useful when you want to estimate a sigmoid by hand.

Notice the symmetry: σ(−2) = 0.119 and σ(2) = 0.881, and 0.119 + 0.881 = 1. This is the identity σ(−z) = 1 − σ(z) in action.

05 Worked example

We will work through two examples. The first uses a single feature so that the arithmetic is easy to follow; the second returns to the loan application from Section 02 and uses two features.

z = −3 + 1.2 × hours

z = −3 + 1.2 × 3 = −3 + 3.6 = 0.6

p = 1 / (1 + e^(−0.6))
  = 1 / (1 + 0.5488)
  = 1 / 1.5488
  = 0.6457

e^1.2 ≈ 3.32

−3 + 1.2 × hours = 0
hours = 3 / 1.2 = 2.5

z = −12 + 0.015 × credit_score + 5 × debt_to_income

z = −12 + 0.015 × 600 + 5 × 0.45
  = −12 + 9.0 + 2.25
  = −0.75

p = σ(−0.75)
  = 1 / (1 + e^(0.75))
  = 1 / (1 + 2.117)
  = 0.321

z = −12 + 0.015 × 750 + 5 × 0.20
  = −12 + 11.25 + 1.00
  = 0.25

p = σ(0.25)
  = 1 / (1 + e^(−0.25))
  = 1 / (1 + 0.7788)
  = 0.562

z = −12 + 0.015 × 820 + 5 × 0.10
  = −12 + 12.30 + 0.50
  = 0.80

p = σ(0.80)
  = 1 / (1 + e^(−0.80))
  = 1 / (1 + 0.4493)
  = 0.690

06 Interactive activity

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z = np.linspace(-6, 6, 400)

fig, ax = plt.subplots(figsize=(7, 4.5))
ax.plot(z, sigmoid(z), linewidth=2.5, color="#1f4e8c")
ax.axhline(0.5, color="#cfdff2", linestyle="--", linewidth=1.2)
ax.axvline(0.0, color="#cfdff2", linestyle="--", linewidth=1.2)
ax.set_xlabel("linear score  z")
ax.set_ylabel("probability  σ(z)")
ax.set_ylim(-0.05, 1.05)
ax.grid(alpha=0.25)
plt.tight_layout()
plt.show()

# Vary the slope coefficient
for beta in [0.3, 0.8, 1.2, 2.0]:
    hours = np.linspace(0, 6, 200)
    z = -3 + beta * hours
    ax.plot(hours, sigmoid(z), label=f"β₁ = {beta}")
ax.legend()

When you vary β₁, you are changing how steeply the probability rises with each hour of tutoring. When you vary β₀, you are shifting the whole curve left or right. Those two operations — steepness and position — are all a logistic regression can do with a single feature, and they correspond exactly to the two coefficients.

07 Self-check questions

Answer each question before revealing the solution. If you get one wrong, re-read the relevant subsection rather than simply moving on.

08 Practice exercises

These exercises build fluency with the mechanics. Work them by hand first; then check your answers computationally.

1. Compute σ(z) to three decimal places for z = −3, −1.5, −0.5, 0, 0.5, 1.5, 3. Verify the symmetry property for each pair.
2. Convert each of the following probabilities to odds and then to log-odds: 0.05, 0.25, 0.50, 0.75, 0.95. What do you notice about the log-odds of complementary probabilities?
3. A model has the form z = −2 + 0.8x. Compute the predicted probability for x = 0, 1, 2, 3, 4, 5. At what value of x does the model predict exactly 0.5?
4. A coefficient is estimated as β = 2.303. What is the odds ratio? What approximate percentage change in odds does a one-unit increase in the feature produce?
5. A model predicts p = 0.92 for a particular case. What is the corresponding linear score z? (Hint: apply the logit.)
6. Two features have coefficients β₁ = 0.4 and β₂ = −1.2. By how much does the log-odds change when x₁ increases by 3 and x₂ decreases by 0.5?
7. Using the loan model from Worked Example 2, compute the predicted probability for an applicant with credit score 680 and debt-to-income 0.35. Would this applicant be approved at a threshold of 0.5? At a threshold of 0.6?
8. Sketch, without plotting software, the curves corresponding to z = −4 + 2x and z = −1 + 0.5x. Describe how they differ in steepness and position, and explain what each difference means in terms of the feature's effect.

09 Problem sets

These problems require more than mechanical calculation. They ask you to reason about model behaviour, interpret results in context, and identify errors in reasoning.

10 Discussion and projects

Discussion questions

1. Logistic regression is often described as a linear model, despite producing a non-linear relationship between features and probability. Is this description misleading? Defend your answer.
2. The sigmoid saturates, meaning that beyond a certain point additional evidence produces almost no change in predicted probability. Is this a desirable property for a decision-making system, or does it risk the model ignoring genuinely important information at the extremes?
3. Probabilities are often collapsed to a binary decision by applying a threshold. In what sense is the threshold a separate modelling choice from the logistic regression itself, and who should be responsible for setting it?
4. The intercept term β₀ is frequently uninterpretable in practical terms, because it corresponds to a case where every feature is zero. Should models include an intercept at all? What would happen if they did not?

Mini-project · Calibration in practice

In this project you will investigate whether a logistic regression's predicted probabilities are actually reliable — a property called calibration.

Calibration is a topic that appears again in Unit 6 (Model Evaluation) and in Unit 19 (Governance & Fairness), where it becomes entangled with the impossibility results for group fairness metrics. A model that is well calibrated overall may still be miscalibrated for a subgroup — and that difference has real consequences.

11 Summary

This tutorial built the core of logistic regression: the machinery that turns a linear score into a probability.

• Linear regression cannot classify. Its output range, constant rate of change, and statistical assumptions are all wrong for a binary outcome.
• The fix is a two-stage model. Compute a linear score z = β₀ + β₁x₁ + … + βₚxₚ, then squash it through the sigmoid σ(z) = 1 / (1 + e^(−z)).
• The sigmoid has the right properties. It is bounded in (0, 1), strictly increasing, symmetric around σ(0) = 0.5, smooth, and saturating at both ends.
• Logistic regression is linear in the log-odds. The inverse of the sigmoid is the logit: log(p / (1 − p)) = z. This is the scale on which the model is genuinely linear.
• Probability, odds and log-odds are three views of one quantity. Their ranges are [0, 1], [0, ∞) and (−∞, ∞) respectively — which is why only the last can be modelled by an unrestricted linear function.
• Coefficients act multiplicatively on the odds. A one-unit increase in xⱼ multiplies the odds by e^(βⱼ). The effect on the probability depends on where on the curve you are.

12 Further reading

Core

• James, Witten, Hastie & Tibshirani — An Introduction to Statistical Learning, Chapter 4. The standard treatment of logistic regression, with clear derivations and worked examples. Section 4.3 covers the material in this tutorial directly.
• scikit-learn documentation — LogisticRegression. Read the "Mathematical formulation" section alongside this tutorial. It presents the model in precisely the terms developed here.
• Hastie, Tibshirani & Friedman — The Elements of Statistical Learning, Section 4.4. A more advanced treatment, including the link to generalised linear models and the derivation of the fitting procedure you will meet in T-04.

Supplementary

• Gelman, Hill & Vehtari — Regression and Other Stories, Chapters 13–14. An accessible treatment of logistic regression with a strong emphasis on interpretation and on presenting results to non-technical audiences.
• Hosmer, Lemeshow & Sturdivant — Applied Logistic Regression. A comprehensive reference covering diagnostics, goodness-of-fit and the practical issues that arise in real applications.
• Niculescu-Mizil & Caruana — "Predicting Good Probabilities with Supervised Learning" (ICML 2005). The classic empirical study of calibration across model families, directly relevant to the mini-project in Section 10.

On the history

• Cramer, J.S. — "The Origins of Logistic Regression" (2002). A short historical account of how the logistic function came to be used for binary outcomes — and why the model carries the confusing name it does.