Unit 8 · Random Forests — COMP328: Fundamentals of Machine Learning

01 Learning Outcomes

Knowledge

Explain what an ensemble is and why averaging many models can beat any single model.
Describe bootstrap sampling and explain why roughly 63% of the original rows appear in each sample.
Explain how bagging reduces variance without materially increasing bias.
Describe random feature selection (max_features) and why it decorrelates the trees.
Distinguish hard voting from soft voting and state when each is appropriate.
Explain what out-of-bag error estimates and why it is a free validation signal.

Skills

Draw a bootstrap sample by hand and identify the out-of-bag rows.
Combine several tree predictions by majority vote and by averaged probability.
Train, tune, and evaluate a RandomForestClassifier with scikit-learn.
Choose sensible values for n_estimators, max_features, and min_samples_leaf.
Interpret and critically question a feature-importance ranking.
Benchmark a forest against the single decision tree from Unit 7.

Professional Competencies

Explain the trade-off between interpretability and predictive performance to a stakeholder.
Recognise that ensembling hides individual decision paths and what that means for accountability.
Detect importance inflation caused by correlated or proxy features.
Report model comparisons honestly, including where the forest does not help.

02 Why This Unit Matters

Unit 7 ended on a weakness: a decision tree is unstable. Change a handful of training rows and the greedy split at the root can flip, producing a completely different tree. Random forests turn that weakness into a strength. Instead of searching for the single best tree, they deliberately grow many imperfect, different trees and average them. The individual trees become noisier, but the ensemble becomes far more reliable.

Big idea

Diversity plus aggregation. If you have many models that are each better than random and that make different mistakes, their majority vote is more accurate than the average of the individual models. Random forests engineer that diversity on purpose — through bootstrapping the rows and sampling the features.

Where you have already met this

The bias–variance trade-off from Unit 6 is the whole story here. A deep tree is low bias and high variance. Bagging attacks the variance term directly, which is why a forest of deep trees is often both more accurate and less prone to overfitting than any of its members.

Common misconception

"More trees always means a better model." Adding trees reduces the variance of the ensemble and improves stability, but returns diminish quickly and the model never becomes perfect. Once the out-of-bag error flattens, extra trees buy you computation time and nothing else.

03 Reading Roadmap

Work through the eight tutorials in order. Tutorials t-02, t-03, and t-04 explain the three mechanisms that make a forest work — bootstrapping, feature sampling, and voting. Do the hand calculations there; they are what make the scikit-learn API feel obvious rather than magical.

t-01

From One Tree to a Forest

Ensemble learning, the wisdom of crowds, and why averaging uncorrelated errors helps.

t-02

Bootstrap Sampling & Bagging

Sampling with replacement, the ~63% rule, out-of-bag rows, and variance reduction.

t-03

Random Feature Selection

Why sampling features at each split decorrelates trees, and how max_features controls it.

t-04

Hard and Soft Voting

Majority vote versus averaged probabilities, tie-breaking, and when the two disagree.

t-05

Out-of-Bag Error

Using the rows each tree never saw as a built-in validation set.

t-06

Forests in scikit-learn

RandomForestClassifier, key hyperparameters, and honest cross-validation.

t-07

Feature Importance

Mean decrease in impurity, permutation importance, and the traps in both.

t-08

Strengths, Limits & Fairness

What forests cannot do, cost and latency, class imbalance, and responsible use.

04 Topics & Key Concepts

Unit 8 syllabus topics mapped to tutorials and outcomes
Topic	What you will be able to do	Tutorials
Ensemble learning	Explain why a diverse group of models can outperform its best member.	t-01, t-04
Bagging	Draw bootstrap samples, identify out-of-bag rows, and explain variance reduction.	t-02, t-05
Voting	Combine predictions by majority vote or averaged probability and justify the choice.	t-04
Feature importance	Rank features, explain what the ranking does and does not mean, and check it with permutation importance.	t-07, t-08
Practical modelling	Fit, tune, and benchmark a random forest against a single decision tree.	t-06, t-07

Terms you must be able to define

Ensemble: A model built by combining the predictions of several base models. The ensemble is usually more accurate and more stable than any individual member.
Bootstrap sample: A sample of the same size as the training set, drawn with replacement. Some rows appear several times; others are omitted entirely.
Bagging: Bootstrap aggregating — training a base model on each bootstrap sample and combining their predictions. Reduces variance, especially for high-variance learners such as deep trees.
Out-of-bag (OOB) rows: The roughly 37% of training rows that do not appear in a given bootstrap sample. They act as a built-in held-out set for that tree.
Random feature selection: At every split, a tree considers only a random subset of the available features, controlled by max_features. This decorrelates the trees, which is what makes averaging effective.
Hard voting: Each tree contributes one class label; the ensemble predicts the class with the most votes.
Soft voting: Each tree contributes a predicted probability; the ensemble averages those probabilities and predicts the most probable class. Usually preferred because it uses more information.
Feature importance (MDI): Mean decrease in impurity — the total weighted impurity reduction a feature contributes across all trees and nodes. Biased toward high-cardinality and correlated features.
Permutation importance: The drop in validation score when a single feature's values are randomly shuffled. Measured on held-out data, so it is generally more trustworthy than MDI.

05 Worked Example: Why the Vote Beats the Average Tree

Suppose we train a small forest of five trees on the student-pass dataset from Unit 7. We then evaluate each tree separately, and the forest as a whole, on five held-out students.

Individual tree predictions on five held-out examples
Example	Tree 1	Tree 2	Tree 3	Tree 4	Tree 5	True label
E1	Pass	Pass	Fail	Pass	Fail	Pass
E2	Fail	Fail	Fail	Pass	Fail	Fail
E3	Pass	Fail	Pass	Pass	Fail	Fail
E4	Fail	Pass	Fail	Fail	Fail	Fail
E5	Pass	Pass	Pass	Fail	Pass	Pass

Step-by-step

Score each tree individually.

Tree 1: 4/5 correct  → 80%
Tree 2: 4/5 correct  → 80%
Tree 3: 3/5 correct  → 60%
Tree 4: 3/5 correct  → 60%
Tree 5: 4/5 correct  → 80%

Mean individual accuracy = (80 + 80 + 60 + 60 + 80) / 5 = 72%

Apply majority voting per example.

E1:  Pass ×3, Fail ×2  →  Pass   ✓ (true: Pass)
E2:  Fail ×4, Pass ×1  →  Fail   ✓ (true: Fail)
E3:  Pass ×3, Fail ×2  →  Pass   ✗ (true: Fail)
E4:  Fail ×4, Pass ×1  →  Fail   ✓ (true: Fail)
E5:  Pass ×4, Fail ×1  →  Pass   ✓ (true: Pass)

Score the ensemble.
```
Forest accuracy = 4/5 = 80%
```
Interpret the result. The forest reaches 80% against a mean individual accuracy of 72%. No single tree was better than 80%, and two were considerably worse — yet the vote matched the best individual tree while being built from weaker components.
Notice where it still fails. E3 is misclassified because three trees made the same mistake. Diversity is not unlimited: trees trained on overlapping bootstrap samples still share blind spots, which is exactly why random feature selection (t-03) matters.

Why not just keep Tree 1?

Because you cannot know in advance which tree will be best on unseen data. On this sample Tree 1 scores 80%, but on a different held-out set Tree 3 might lead. The ensemble does not require you to guess correctly — it hedges. That hedging is the entire value of bagging.

A note on bootstrap samples

Each tree above was trained on its own bootstrap sample. For a dataset of n rows, the probability that a given row is not drawn in one sampling step is 1 − 1/n. Across n independent draws:

P(row omitted) = (1 − 1/n)ⁿ  →  e⁻¹ ≈ 0.368

So each bootstrap sample contains roughly 63.2% of the distinct
original rows, and leaves about 36.8% out-of-bag.

Those out-of-bag rows are not wasted — they become a validation set that comes free with the algorithm. Tutorial t-05 shows how to use them.

06 Learning Activities

Activity 8.1 — Draw a bootstrap sample by hand

Number the eight students from the Unit 7 dataset 1–8. Using a random number generator (or a die), draw eight numbers with replacement. Record your sample, then list which student IDs are out-of-bag. Repeat three times and compare how many distinct rows you get each time against the predicted value of 63.2%.

Activity 8.2 — Hard vote versus soft vote

Five trees produce the following predicted probabilities of Pass for one student: 0.90, 0.85, 0.45, 0.40, 0.10.

Compute the hard-voting prediction (threshold each at 0.5, then take the majority).
Compute the soft-voting prediction (average the probabilities, then threshold at 0.5).
The two disagree. Which would you trust, and what does the disagreement tell you about the individual trees?

Activity 8.3 — Forest versus single tree

In the notebook, fit the Unit 7 decision tree and a random forest on the same dataset with identical train/test splits. Record training accuracy, test accuracy, and 5-fold cross-validated accuracy for both. Write three sentences describing where the forest helped and whether the cost in training time and interpretability was justified.

Notebook

u08-random-forests.ipynb Guided notebook — grow forests of increasing size, track OOB error, compare hard and soft voting, and inspect feature importances. Open

The core of the notebook is a compact, readable scikit-learn workflow:

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

rf = RandomForestClassifier(
    n_estimators=300,        # number of trees
    max_features="sqrt",     # features considered per split (classification default)
    min_samples_leaf=2,      # smooths individual trees
    oob_score=True,          # free validation estimate
    random_state=42,
    n_jobs=-1,               # use all available cores
)

rf.fit(X_train, y_train)

print("OOB score: ", round(rf.oob_score_, 3))
print("CV accuracy:", round(cross_val_score(rf, X_train, y_train, cv=5).mean(), 3))
print("Importances:", dict(zip(feature_names, rf.feature_importances_.round(3))))

Watch the tree count

Plot OOB error against n_estimators from 1 to 300. The curve drops steeply at first and then flattens. Beyond the flattening point you are paying for computation without buying accuracy — a useful thing to show a stakeholder who asks "why not 10,000 trees?"

07 Self-Check

Answer these before moving on. If any feel uncertain, revisit the linked tutorial.

08 Assessment Activities

3%

Quiz 8 · Random Forests

12 questions covering ensemble reasoning, bootstrap sampling, OOB error, hard versus soft voting, max_features, and feature-importance interpretation. Unlimited attempts; highest score counts.

10%

Assignment 3 · Classification Analysis

The same assignment introduced in Unit 7. Extend it by benchmarking your decision tree against a random forest, reporting both sets of metrics, and justifying which model you would deploy.

Quiz 8 Random Forests — 12 questions, auto-graded Due end of week 8
Assignment 3 Classification Analysis — tree vs forest comparison Due week 8
Capstone Ensemble model for your capstone problem, with importance analysis End of course

Tip for Assignment 3

Report the cost of the forest as well as its accuracy — training time, inference latency, and the fact that you can no longer show a stakeholder a single readable tree. A model recommendation that ignores those costs is incomplete, however good the numbers look.

09 Summary

What you should take away

A random forest is an ensemble of decision trees. It trades the interpretability of a single tree for stability and predictive accuracy.
Diversity is engineered in two ways: bootstrap sampling of the rows (bagging) and random feature selection at each split. Both are essential — bagging alone leaves the trees correlated.
Predictions are combined by majority vote or by averaging predicted probabilities. Soft voting usually wins because it uses confidence information that hard voting throws away.
Out-of-bag error gives a validation estimate for free, without carving out a separate hold-out set.
Feature importance is useful but biased — it favours high-cardinality and correlated features. Confirm rankings with permutation importance on held-out data before acting on them.
More trees reduce variance but cannot fix a biased or unrepresentative dataset. Ensembling is not a substitute for good data.

Next up

Unit 9, Support Vector Machines, takes a completely different route to the same problem: instead of averaging many simple models, it finds the single optimal boundary that separates the classes with the widest possible margin — and uses kernels to handle data that no straight line can divide.

Random Forests