Unit 15 — Deep Learning

01 Unit at a glance

Where this unit sits

In Unit 14 you derived forward propagation and backpropagation for a small network. Here you scale that machinery up. Depth is what turns a neural network into a deep network, and depth brings three new problems: gradients that vanish, training that destabilises, and models that memorise instead of generalise. Unit 15 is a tour of the solutions — better optimisers, careful initialisation, and regularisation. Unit 16 then applies these ideas to images.

Unit 15 summary
Module	Module 5 — Neural Networks and Deep Learning
Estimated study time	5 – 7 hours (9 tutorials, quiz, practical notebook and tuning exercises)
Format	9 tutorial pages · 1 unit quiz · 1 practical notebook · 1 assignment component
Prerequisites	Unit 14 (neurons, activations, forward propagation, backpropagation, gradient descent) and Unit 6 (model evaluation, bias–variance tradeoff, cross-validation). Familiarity with NumPy arrays is assumed.
Tools	Python, NumPy, scikit-learn, Jupyter, Matplotlib

Deep networks Optimisation Regularisation Overfitting Applied ML

02 Learning outcomes

By the end of Unit 15 you should be able to do all of the following. These outcomes are assessed in the unit quiz, in Assignment 5, and again in the capstone project.

Knowledge — explain

Explain why adding hidden layers increases a network's expressive power beyond a single linear boundary.
Describe what representation learning means and why early layers capture different features from later layers.
Compare SGD, momentum, RMSProp and Adam and state when each is preferred.
Explain the vanishing and exploding gradient problems in terms of repeated multiplication of derivatives.
Distinguish L1, L2, weight decay, dropout and early stopping as different constraints on the model.

Skills — apply

Select a sensible weight initialisation scheme (Xavier / He) for a given activation function.
Add and tune dropout and L2 regularisation in a working network.
Choose an optimiser, batch size and learning-rate schedule for a training run.
Read training and validation curves to diagnose underfitting, overfitting and unstable training.

Professional competencies — evaluate

Justify the added complexity of a deep model against a simpler baseline on the same data.
Report training cost, tuning effort and environmental footprint alongside accuracy.
Recognise that deeper models intensify the interpretability and accountability concerns introduced in Unit 18.

03 Reading roadmap

Work through the nine tutorials in order. Tutorials 01–02 build the conceptual case for depth; 03–05 cover the mechanics that make deep training possible; 06–07 cover regularisation; 08–09 put everything into practice.

Tutorial 01

From shallow to deep

What changes when a network gains depth, and why a single hidden layer is sometimes not enough.

Tutorial 02

Hidden layers & representation learning

Layers as learned feature transformations, and hierarchical feature composition.

Tutorial 03

Optimisation: SGD, momentum, Adam

Beyond plain gradient descent — how modern optimisers smooth and scale updates.

Tutorial 04

Vanishing & exploding gradients

Why deep gradients misbehave, and the diagnostic signals that reveal it.

Tutorial 05

Weight initialisation

Zero, random, Xavier and He initialisation — and why the starting point decides the outcome.

Tutorial 06

Regularisation: L1, L2 and weight decay

Penalising complexity, and the difference between a penalty term and a decay step.

Tutorial 07

Dropout & early stopping

Stochastic regularisation and the simplest way to stop training at the right moment.

Tutorial 08

Building a deep network in code

A complete worked example with MLPClassifier, from split to scored evaluation.

Tutorial 09

Diagnosing deep training

Learning curves, instability, capacity tuning and a first look at transfer learning.

Suggested pacing
Session	Tutorials	Focus	Time
1	01 – 02	Why depth helps and what hidden layers learn	75 min
2	03 – 05	Optimisers, gradient pathologies, initialisation	120 min
3	06 – 07	Regularisation and dropout in practice	90 min
4	08 – 09	Implementation, diagnosis and tuning	105 min
Total guided study			≈ 6.5 h

04 Learning activities

Deep learning is learned by training models badly and then fixing them. Complete each activity as you reach the corresponding tutorial.

Activity 15.1 — Depth vs width

After Tutorial 01, train the same classifier twice: once as a wide network with one hidden layer of 128 units, once as a deep network with four hidden layers of 32 units. Compare parameter counts and validation accuracy.

Activity 15.2 — Visualise a learned representation

After Tutorial 02, extract the activations of the first hidden layer on a 2-D dataset and plot them. Describe in two sentences what the layer appears to have separated.

Activity 15.3 — Optimiser bake-off

After Tutorial 03, train identical architectures with SGD, SGD + momentum, and Adam for the same number of epochs. Plot the loss curves together and record the epochs needed to reach a target loss.

Activity 15.4 — Break the gradients

After Tutorial 04, deliberately initialise weights with a very large scale and observe the loss. Then try a very small scale with a sigmoid activation. Record what happens in both cases and explain why.

Activity 15.5 — Regularisation grid

After Tutorials 06 and 07, run a small grid over L2 penalty strength and dropout rate. Produce a table of training accuracy, validation accuracy and the gap between them.

Activity 15.6 — Early stopping by hand

After Tutorial 07, train for many more epochs than necessary while recording validation loss each epoch. Identify the epoch where early stopping would have triggered and compare that model's test score with the final one.

Unit 15 practical notebook All six activities in one runnable notebook — unit-15-practical.ipynb

05 Assessment activities

10 q

Unit 15 quiz

Multiple choice and short answer covering hidden layers, optimisers, gradient pathologies, initialisation and regularisation. Unlimited attempts; highest score counts.

A5

Assignment 5 — Neural Network Modeling

The deep-learning component of Assignment 5: justify your architecture, tune the optimiser and regularisation, and compare against a logistic regression baseline.

Cap

Capstone contribution

Deep models must be defended on cost and interpretability. If you use one in your capstone, document the tuning effort and the limits of your explanation.

How Unit 15 feeds Assignment 5

A5 · Part 2 Architecture design — depth, width and activation choices Instructions →
A5 · Part 2 Optimiser, learning rate and batch size selection Instructions →
A5 · Part 3 Regularisation and early stopping — evidence of reduced overfitting Rubric →
A5 · Part 4 Reflection: training cost, tuning effort and interpretability Submit →

06 Key terms

Deep network: A neural network with more than one hidden layer, allowing hierarchical feature composition.
Representation learning: Learning the features themselves from data, rather than relying on features chosen by a human.
Stochastic gradient descent (SGD): Gradient descent using a randomly sampled mini-batch of examples to estimate the gradient at each step.
Momentum: An optimiser that accumulates a running average of past gradients, smoothing updates and accelerating through flat regions.
Adam: An adaptive optimiser that maintains per-parameter learning rates using estimates of the first and second moments of the gradient.
Vanishing gradient: The tendency for gradients to shrink towards zero as they are propagated back through many layers, stalling learning in early layers.
Weight initialisation: The scheme used to set starting weights — Xavier/Glorot for sigmoid and tanh, He initialisation for ReLU.
Regularisation: Any technique that constrains model complexity to improve generalisation, such as L1, L2, dropout or early stopping.
Dropout: Randomly deactivating a proportion of units during each training step, which prevents co-adaptation of features.
Early stopping: Halting training when validation performance stops improving, thereby limiting overfitting.

07 Summary

What you have covered

Unit 15 turned the small network of Unit 14 into a deep one. You saw that depth buys expressive power because each layer learns a transformation of the previous layer's representation, letting the network build complex features out of simple ones. You then met the practical cost of that power: gradients that vanish or explode, training that becomes sensitive to initialisation, and models that happily memorise a training set.

The remedies form a toolbox you will reuse for the rest of the course — adaptive optimisers such as momentum and Adam, principled initialisation such as Xavier and He, and regularisation through L1/L2 penalties, weight decay, dropout and early stopping. None of these is a universal answer; each trades bias against variance in a different way, and choosing among them is an empirical exercise guided by validation curves.

Before you move on

You should be able to look at a pair of training and validation loss curves and state whether the model is underfitting, overfitting, or training unstably — and name at least two interventions for each case.

Common misconception

Dropout is not a free win. At evaluation time all units are active, so the scale of activations changes; frameworks handle this correction automatically, but a dropout rate set too high will simply underfit the model.

Watch out

Never tune hyperparameters against the test set. Split off a validation set (or use cross-validation on the training data) and touch the test set once, at the end.