Unit 11 — Dimensionality Reduction | COMP328: Fundamentals of Machine Learning

01 Learning outcomes

By the end of Unit 11 you should be able to do the following. Each outcome is assessed somewhere — the mapping is given in Section 05.

Knowledge — explain

Describe the curse of dimensionality and why distance metrics become less informative as the number of features grows.
Distinguish feature selection from feature extraction, and state when each is appropriate.
Explain what a principal component represents geometrically and statistically.
Define explained variance and interpret a scree plot.
Compare PCA, t-SNE, and UMAP in terms of what they optimise and what their outputs can be used for.

Skills — apply

Standardise a feature matrix appropriately before applying PCA.
Fit a PCA object, read its explained-variance ratios, and choose a number of components.
Compute and interpret PCA loadings to understand what each component means.
Produce 2-D t-SNE and UMAP embeddings of a high-dimensional dataset.
Build a scikit-learn Pipeline that chains scaling, reduction, and a downstream estimator.
Quantify how much downstream model performance changes when you reduce dimensions.

Professional competencies — evaluate

Recognise when a t-SNE plot is being over-interpreted, and say so.
Report the information cost of compression honestly rather than presenting it as free.
Document reduction choices so that a pipeline is reproducible.
Assess whether reduction is helping or quietly destroying signal.

02 Why dimensionality reduction matters

Adding features feels like adding information. Usually it is. But past a certain point, each new feature makes the data sparser in the space the algorithm actually sees. If you fix the number of observations and keep adding dimensions, the average distance between points grows, and the ratio between the nearest and farthest neighbour approaches 1. Everything starts to look equally far from everything else. Clustering, k-nearest neighbours, and kernel methods all quietly stop working.

There is a second, blunter problem: you cannot see the data. A dataset with 784 features — a single 28×28 greyscale image — is invisible to you. Without a way to project it into two dimensions, you are debugging blind. PCA, t-SNE, and UMAP are how you get your eyes back.

A typical scree plot. The first component captures 42% of the total variance, the second another 23%, the third 12%. After the third component the bars flatten out — the "elbow". Retaining the first three components keeps 77% of the variance while discarding half the original features.

Common misconception

Dimensionality reduction is not compression without cost. When you keep three components out of twenty, you are throwing away 23% of the variance in this example — and some of that discarded variance may be exactly the signal your downstream model needed. Reduction is a trade: fewer dimensions and less noise, in exchange for some information loss. Always measure what the trade cost you.

03 Reading roadmap

The nine tutorials below are designed to be read in order. T-01 and T-02 establish the problem; T-03 to T-05 develop PCA in depth; T-06 and T-07 cover nonlinear and neural approaches; T-08 and T-09 put everything into practice.

T-01

The curse of dimensionality

Why distance-based methods degrade as features multiply, and what "sparse data" really means.

T-02

Feature selection vs feature extraction

Filter, wrapper, and embedded selection methods; when to drop columns and when to combine them.

T-03

PCA — intuition

Finding the directions of maximum variance; projection, reconstruction, and what a component means.

T-04

PCA — mathematics

Covariance matrices, eigenvectors, eigenvalues, and the singular value decomposition behind the scenes.

T-05

Choosing the number of components

Scree plots, cumulative explained variance, Kaiser's rule, and the reconstruction-error approach.

T-06

t-SNE and UMAP

Nonlinear embeddings for visualisation: perplexity, neighbourhoods, and why the axes mean nothing.

T-07

Autoencoders preview

Neural networks that learn compressed representations — a bridge to Module 5.

T-08

Dimensionality reduction in scikit-learn

Building pipelines, avoiding data leakage, and measuring downstream impact.

T-09

Case study: visualising high-dimensional data

Applying PCA, t-SNE, and UMAP to a real image dataset and reporting what each reveals.

Study tip

T-04 is the mathematical heart of this unit. If eigenvectors and eigenvalues are unfamiliar, work through the two-by-two numerical example in that tutorial completely before moving on — by hand, with a pen. Every later result in PCA follows from that one calculation.

04 Learning activities

Each tutorial contains a worked example and a self-check. The activities below are the ones that require you to write and run code.

Activity 11.1 — PCA by hand

Using the four-point dataset from T-04, centre the data, compute the 2×2 covariance matrix, and find its eigenvalues and eigenvectors by hand. Verify your answer with np.linalg.eig. Then project the points onto the first component and confirm that the projected variance equals the eigenvalue.

Activity 11.2 — How many components?

Load the breast cancer dataset from scikit-learn (30 features). Fit PCA with all components and plot the scree plot and cumulative explained variance. Determine the smallest number of components that retains 95% of the variance. Then train a logistic regression on the reduced data and compare its accuracy to the full-feature baseline.

Activity 11.3 — Seeing is believing

Take a 2,000-image subset of the digits dataset. Produce three 2-D visualisations: PCA, t-SNE with perplexity=30, and UMAP. Colour the points by true digit label. For each plot, note which digit classes separate cleanly and which overlap. Then re-run t-SNE with perplexity=5 and perplexity=100 and describe how the picture changes.

unit-11-dimensionality-reduction.ipynb Starter notebook containing datasets, plotting helpers, and scaffolded cells for activities 11.1–11.3. Open workflow

05 Assessment activities

10%

Unit 11 quiz

12 questions covering the curse of dimensionality, PCA mechanics, explained variance, and the appropriate use of t-SNE and UMAP. Unlimited attempts; highest score counts.

15%

Assignment 4

Unsupervised learning. Your submission must include a dimensionality-reduction stage with a justified choice of method and number of components.

—

Self-checks

Embedded in every tutorial. Ungraded, but they are the fastest way to find out whether you have actually understood the previous section.

Outcome-to-assessment mapping

Where each learning outcome is assessed
Outcome	Quiz	Assignment 4	Capstone
Explain the curse of dimensionality	Q1–Q3	Report §1	Methodology
Distinguish selection from extraction	Q4–Q5	Report §2	Methodology
Apply PCA and interpret components	Q6–Q9	Code + results	Model building
Choose a number of components	Q10	Report §3	Evaluation
Use t-SNE / UMAP appropriately	Q11–Q12	Figures	Interpretation

Quiz 11 Dimensionality reduction — 12 questions, 30 minutes Revise

Assignment 4 Unsupervised Learning: clustering, dimensionality reduction, and association rules Open brief

06 Key terms

Curse of dimensionality: The set of problems that arise when data becomes sparse in high-dimensional space — distance concentration, exponential data requirements, and degraded model performance.
Feature selection: Choosing a subset of the original features to keep, and discarding the rest. The remaining features are unchanged and still interpretable.
Feature extraction: Creating new features as combinations of the originals — for example, the principal components produced by PCA.
Principal component: A direction in feature space along which the projected data has maximum variance, subject to being orthogonal to all previously found components.
Explained variance ratio: The fraction of the total variance captured by a given principal component. The ratios sum to 1 across all components.
Scree plot: A bar chart of the variance explained by each successive component, used to locate the "elbow" where additional components stop adding much.
Loading: The weight of each original feature in a principal component. Loadings are how you interpret what a component represents.
t-SNE: t-distributed Stochastic Neighbour Embedding — a nonlinear method for visualisation that preserves local neighbourhood structure.
Perplexity: t-SNE's main tuning parameter, roughly controlling the effective number of neighbours each point considers. Typical values are 5–50.
UMAP: Uniform Manifold Approximation and Projection — a nonlinear embedding method that is often faster than t-SNE and better at preserving global structure.
Reconstruction error: The distance between the original data and its reconstruction from the reduced representation. Used both to choose components and to train autoencoders.

07 Summary

What you should take away

More features is not automatically better. Beyond a point, each additional dimension makes the data sparser and distance-based methods less reliable. This is the curse of dimensionality.
Feature selection keeps original, interpretable columns. Feature extraction creates new, often uninterpretable ones. Both reduce dimensionality; they are not interchangeable.
PCA finds orthogonal directions of maximum variance. It is linear, unsupervised, fast, and deterministic.
Always standardise before PCA unless all features share the same units and you deliberately want to preserve scale differences.
Choose the number of components from the scree plot and cumulative explained variance — not from a fixed rule. Validate the choice against downstream performance.
t-SNE and UMAP are for visualisation. Their axes and absolute distances are not meaningful. Do not feed their output into a downstream model.
Reduction is a trade, not a free lunch. Measure what you lost.

Ready to move on?

Once you are comfortable with the tutorials and have scored at least 80% on the unit quiz, continue to Unit 12 — Association Learning, where you will learn how to discover co-occurrence patterns and if–then rules in transactional data.

08 Frequently asked questions

How much linear algebra do I need for PCA?

More than for clustering, but still manageable. You need to understand what a matrix–vector product does geometrically, what a covariance matrix measures, and what it means for a vector to be an eigenvector of one. T-04 develops all of this from first principles using two-by-two examples. If you can multiply a 2×2 matrix by a 2-vector and solve a quadratic equation, you have enough to follow it.

Should I reduce dimensions before clustering?

Often yes, but not always, and never automatically. Reduction helps when many features are correlated or noisy, because it removes redundancy that inflates distances. It hurts when the cluster structure lives in a direction of low variance — PCA will discard exactly the axis that separates your clusters. Always compare clustering results on the full feature set and on the reduced set before committing.

PCA or t-SNE for my assignment figures?

Use both, for different purposes. PCA gives you an interpretable linear projection — you can say what each axis means and how much variance it captures. t-SNE gives you a clearer picture of local neighbourhood structure but no interpretable axes. A good report shows a PCA plot with explained-variance labels and a t-SNE plot for visual separation, and is explicit about what each one does and does not demonstrate.

Why does my t-SNE plot look completely different each time I run it?

t-SNE is stochastic — it starts from a random initialisation and optimises a non-convex objective. Different runs land in different local optima. Set random_state for reproducibility within a report, but be aware that the global arrangement of clusters is not stable across seeds. Only the local neighbourhood relationships should be trusted, and only if they persist across several runs with different perplexity values.

Is UMAP always better than t-SNE?

No. UMAP is generally faster on large datasets and tends to preserve more global structure, which makes it attractive for exploratory work. But it has its own tuning parameters (n_neighbors, min_dist) and its own failure modes, and the theoretical assumptions behind it are more involved. For a course assignment, either is acceptable provided you justify the choice and report the parameter settings you used.