Class Imbalance and Thresholds

04 Visual illustration

Figure 1 shows the confusion matrix for the fraud model at the default threshold of 0.5. Every metric in this tutorial is derived from the four cells in this diagram.

Figure 2 plots precision and recall as the threshold moves from 0.9 (high, conservative) down to 0.1 (low, aggressive). The curve shows the fundamental trade-off: you cannot increase recall without sacrificing precision.

The gap between the curve and the baseline is what the model adds. Because the baseline is so low — only 1% — even the poor precision at low thresholds represents real signal. This is why PR curves are essential under severe imbalance: they make the model's actual contribution visible, which the ROC curve does not.

05 Worked example

We work through the fraud model in full, Computing every metric at every threshold, and then selecting a threshold by expected cost.

Cost(t) = 800 · FN(t) + 50 · FP(t)

06 Interactive activity

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score

# Imbalanced data: 2% positive
X, y = make_classification(n_samples=20000, n_features=6, n_informative=4,
                           n_redundant=0, weights=[0.98, 0.02],
                           class_sep=1.0, flip_y=0.01, random_state=0)

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
                                          stratify=y, random_state=0)

clf = LogisticRegression(penalty=None, max_iter=5000).fit(X_tr, y_tr)
probs = clf.predict_proba(X_te)[:, 1]

thresholds = np.arange(0.05, 0.96, 0.05)
rows = []
for t in thresholds:
    preds = (probs >= t).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_te, preds).ravel()
    rows.append({
        "t": t,
        "TP": tp, "FP": fp, "FN": fn, "TN": tn,
        "precision": precision_score(y_te, preds, zero_division=0),
        "recall": recall_score(y_te, preds, zero_division=0),
        "f1": f1_score(y_te, preds, zero_division=0),
    })

import pandas as pd
df = pd.DataFrame(rows)
print(df.round(3))

# Cost-optimal threshold
C_FN, C_FP = 100, 10
df["cost"] = C_FN * df["FN"] + C_FP * df["FP"]
best = df.loc[df["cost"].idxmin()]
print(f"\nCost-optimal threshold: {best['t']:.2f}, cost = ${best['cost']:,.0f}")

# PR curve
fig, ax = plt.subplots(figsize=(7.5, 5))
ax.plot(df["recall"], df["precision"], marker="o", linewidth=2.2,
        color="#1f4e8c")
prevalence = y_te.mean()
ax.axhline(prevalence, color="#9b2c1c", linestyle="--",
           label=f"baseline (prevalence = {prevalence:.3f})")
ax.set_xlabel("recall")
ax.set_ylabel("precision")
ax.set_title("Precision-recall curve")
ax.legend()
ax.grid(alpha=0.25)
plt.tight_layout()
plt.show()

Two results usually surprise students. First, the precision-recall curve may not be monotonic — with a small test set and few positives, precision can be noisy. Second, refitting with class_weight='balanced' tends to shift the whole curve up and to the right, but the shape is similar. The weights change the probabilities, not the ranking — and since the PR curve only depends on the ranking, the curve is essentially the same curve with a different threshold mapping.

07 Self-check questions

Answer each question before revealing the solution.

08 Practice exercises

Work these by hand where possible, then verify computationally.

1. A test set has 500 positives and 9,500 negatives. A model predicts positive for 1,000 cases, of which 300 are true positives. Compute the confusion matrix, precision, recall, specificity and F1.
2. A model has precision 0.25 and recall 0.60. Compute F1. Compute F2 (which weights recall twice as heavily). Which metric is higher, and why?
3. A dataset has 0.5% positive rate. A model with perfect ranking and a threshold of 0.5 achieves precision 0.15 and recall 0.90. Explain why the precision is so low even though the ranking is perfect.
4. A model produces the following counts at three thresholds:
   • t = 0.7: TP = 200, FP = 100, FN = 800, TN = 98,900
   • t = 0.5: TP = 500, FP = 900, FN = 500, TN = 98,100
   • t = 0.3: TP = 800, FP = 3,000, FN = 200, TN = 96,000
   Compute precision, recall, F1 and accuracy at each threshold. Which threshold maximises F1? Which maximises accuracy?
5. Using the same counts as exercise 4, compute expected cost at each threshold for C_FN = $500 and C_FP = $20. Which threshold minimises cost?
6. For the same counts, compute expected cost for C_FN = $100 and C_FP = $50. How does the optimal threshold change?
7. A model achieves ROC-AUC = 0.95 and PR-AUC = 0.10. Interpret both numbers. Which one should you report to a stakeholder concerned about false alarms?
8. A model is trained with class_weight='balanced' on data with a 2% positive rate. It predicts p = 0.6 for a particular case. Should this be interpreted as a 60% chance of a positive outcome? Explain.

09 Problem sets

These problems require reasoning about the trade-offs rather than computation.

10 Discussion and projects

Discussion questions

1. The cost-optimal threshold is a single number that encodes a value judgement — how much a false negative is worth relative to a false positive. Who should be responsible for setting that judgement in a medical screening programme? In a fraud detection system?
2. A model with a fixed threshold produces the same decision rule for every case. Would it ever be better to vary the threshold by context — for example, using a lower threshold for high-risk customers? What are the fairness implications of doing so?
3. Class weighting, resampling and threshold adjustment all change the trade-off between precision and recall. Are these three approaches interchangeable, or does each have a distinct role? Give an example where one is clearly the right choice.
4. The accuracy metric fails under imbalance because it treats both kinds of error equally. Is there any analogous failure for precision, recall or F1? What assumptions does each of these metrics make about what matters?

Mini-project · Full evaluation of an imbalanced classifier

In this project you will take an imbalanced dataset of your choice and produce a complete evaluation, including a recommendation for a specific threshold and an honest discussion of the limitations. This is the kind of analysis that would accompany a real deployment decision.

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import (confusion_matrix, precision_score,
                             recall_score, f1_score, roc_auc_score,
                             average_precision_score)
from sklearn.calibration import CalibratedClassifierCV

# Fit the model
clf = LogisticRegression(penalty=None, max_iter=5000).fit(X_tr, y_tr)
probs = clf.predict_proba(X_te)[:, 1]

# Overall discrimination
print(f"ROC-AUC: {roc_auc_score(y_te, probs):.3f}")
print(f"PR-AUC:  {average_precision_score(y_te, probs):.3f}")

# Threshold sweep with all metrics
rows = []
for t in np.arange(0.05, 0.96, 0.05):
    preds = (probs >= t).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_te, preds).ravel()
    rows.append({
        "t": round(t, 2),
        "TP": tp, "FP": fp, "FN": fn, "TN": tn,
        "precision": precision_score(y_te, preds, zero_division=0),
        "recall": recall_score(y_te, preds, zero_division=0),
        "f1": f1_score(y_te, preds, zero_division=0),
        "accuracy": (tp + tn) / len(y_te),
    })
df = pd.DataFrame(rows)
print(df.round(3).to_string(index=False))

# Calibrate the model for probability interpretation
calibrated = CalibratedClassifierCV(clf, method="isotonic", cv=5)
calibrated.fit(X_tr, y_tr)
cal_probs = calibrated.predict_proba(X_te)[:, 1]
print(f"\nRaw mean probability:       {probs.mean():.4f}")
print(f"Calibrated mean probability: {cal_probs.mean():.4f}")
print(f"Actual prevalence:           {y_te.mean():.4f}")

11 Summary

This tutorial addressed the gap between a probability and a decision — the point at which most classification projects succeed or fail.

• Class imbalance breaks accuracy. With a rare positive class, the trivial all-negative classifier achieves high accuracy while doing nothing useful. Accuracy cannot distinguish between a good model and a useless one.
• The confusion matrix is the foundation. Every useful metric derives from the four counts: TP, FP, FN and TN. Report it before reporting any summary number.
• Precision and recall measure the two kinds of success. Precision is the purity of positive predictions (the cost of false alarms); recall is the coverage of the positive class (the cost of missed cases).
• F1 balances the two. It is the harmonic mean, which is dominated by the smaller of precision and recall — so it is high only when both are reasonably high. The F-beta score allows weighting one more heavily.
• Precision and recall trade off. Lowering the threshold raises recall and lowers precision; raising the threshold does the reverse. This is a property of the decision rule, not the model.
• PR curves are better than ROC curves under severe imbalance. The ROC curve's false positive rate is diluted by the large negative class, while precision is not. When the positive class is rare, plot the PR curve.
• Thresholds should be chosen by cost, not by convention. When the costs of the two errors differ — which is almost always — the optimal threshold is not 0.5. Compute the expected cost at each threshold and pick the minimum.
• Class weights and resampling change the fitting, not just the decision. They can improve the model's ability to separate the classes, but they distort the predicted probabilities.
• Calibration matters when probabilities are used directly. If you need p = 0.7 to mean "70% chance," check and correct the calibration. If you only need the ranking, calibration is unnecessary.
• The threshold is a decision, not a model parameter. It should be documented, reviewed when the cost structure changes, and owned by whoever is accountable for the consequences.

12 Further reading

Core

• Provost & Fawcett — Data Science for Business, Chapter 7. The clearest treatment of the threshold as a business decision, with extensive discussion of cost-benefit analysis and the precision-recall trade-off.
• He & Garcia — "Learning from Imbalanced Data" (IEEE TKDE, 2009). A comprehensive survey of the field, covering metrics, resampling methods and algorithmic approaches to imbalance.
• Saito & Rehmsmeier — "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets" (PLOS ONE, 2015). The definitive empirical demonstration of why PR curves are preferable under imbalance, with clear figures.

Supplementary

• Chawla, Bowyer, Hall & Kegelmeyer — "SMOTE: Synthetic Minority Over-sampling Technique" (JAIR, 2002). The original SMOTE paper. Read it for the motivation and the algorithm; use the imblearn implementation in practice.
• King & Zeng — "Logistic Regression in Rare Events Data" (Political Analysis, 2001). A widely cited treatment of the specific difficulties of fitting logistic regression when the positive class is rare, including the bias in the estimated intercept.
• Niculescu-Mizil & Caruana — "Predicting Good Probabilities with Supervised Learning" (ICML, 2005). An empirical study of calibration across model families, including logistic regression, SVMs and boosted trees. Useful background for the calibration section.

On metrics and evaluation

• Fawcett — "An Introduction to ROC Analysis" (Pattern Recognition Letters, 2006). The classic tutorial on ROC curves. Read it alongside Saito & Rehmsmeier for a balanced view.
• Brodersen, Ong, Stephan & Buhmann — "The Balanced Accuracy and Its Posterior Distribution" (ICPR, 2010). Introduces balanced accuracy and explains why it is a better default than raw accuracy under imbalance.
• Chicco & Jurman — "The Advantages of the Matthews Correlation Coefficient (MCC) over F1 Score and Accuracy in Binary Classification Evaluation" (BMC Genomics, 2020). Makes the case for MCC as a single-number summary that is robust to imbalance and to class swapping.