COMP326Data Science Foundations

Module 2 · Unit 5 · 9 tutorials · Week 5

Data Cleaning

Diagnosing and repairing quality problems: missing values, duplicates, inconsistencies, outliers and errors. The decisions that determine whether your analysis can be trusted.

Start t-01 Module overview

Overview

Real datasets are almost never clean. Values are missing, records are duplicated, formats are inconsistent, and some observations are simply wrong. This unit teaches you to diagnose each class of problem and to make defensible decisions about how to handle it.

The emphasis is on judgement rather than technique. There is rarely one correct approach to a missing value — but there are approaches that can be justified and documented, and approaches that cannot.

Learning outcomes

By the end of this unit you will be able to

  • Assess the quality of a dataset against standard criteria.
  • Diagnose and handle missing data appropriately.
  • Identify and resolve duplicate records.
  • Detect outliers and decide whether to keep or remove them.
  • Document every cleaning decision and its rationale.

Tutorial sequence

t-01

Data Quality

Dimensions of quality: completeness, accuracy, consistency, timeliness.

 t-02

Missing Data

Mechanisms, patterns and defensible strategies for handling gaps.

 t-03

Duplicate Records

Exact and fuzzy duplicates, and how to identify and resolve them.

 t-04

Outliers

Detection methods and the judgement call about whether to remove.

 t-05

Data Standardization

Units, formats, categories — making values comparable across sources.

 t-06

Error Detection

Range checks, consistency rules and plausibility tests.

 t-07

Cleaning Workflow

A reproducible process from assessment through to validation.

 t-08

Practical Lab

Hands-on cleaning of a messy real-world dataset.

 t-09

Unit Summary

Consolidated key points and a forward look to Unit 6.

Key concepts

Data quality dimensions
The attributes — completeness, accuracy, consistency, timeliness, validity — against which a dataset is assessed.
Missing data mechanisms
MCAR, MAR and MNAR — the three ways missingness can arise, each requiring a different handling strategy.
Outlier
An observation that is unusually distant from the rest of the data. Whether it should be removed depends on whether it is an error or a genuine extreme value.
Cleaning audit trail
The documented record of every cleaning decision, its justification and its effect on the dataset.

Assessment alignment

Directly assessed in Assignment 2

Assignment 2 (Data Cleaning and Preparation) requires you to clean a supplied dataset and to justify every decision in a written cleaning report. The techniques in this unit are the substance of that assignment.

👤