Unit 4: Data Acquisition

Overview

Before you can analyse data you have to obtain it. This unit surveys the main categories of data source — files, databases, web APIs, open repositories and web pages — and the practical skills needed to retrieve data from each.

You will also learn how to read the common data formats, how to combine datasets from different sources, and how to document where your data came from so that your analysis can be reproduced.

Learning outcomes

By the end of this unit you will be able to

Identify appropriate data sources for a given analytical question.
Retrieve data from open government portals and web APIs.
Read and write CSV, JSON and tabular file formats.
Integrate datasets from multiple sources into a single table.
Document the provenance of a dataset.

Lesson sequence

t-01

Data Sources

The main categories of data source and how to choose between them.

t-02

Open Government Data

National statistics portals, licensing and the data they publish.

t-03

Web APIs

REST principles, authentication, pagination and rate limits.

t-04

Data Formats

An overview of tabular, hierarchical and binary data formats.

t-05

CSV

Reading, writing and the pitfalls of comma-separated values.

t-06

JSON

Nested structures, and how to flatten JSON into tabular form.

t-07

Data Integration

Joining, concatenating and reconciling multiple sources.

t-08

Unit Summary

Consolidated key points and a forward look to Unit 5.

Key concepts

API: Application Programming Interface — a documented way for programs to request data or services from a remote system.
REST: A widely used style of web API based on stateless HTTP requests to URLs representing resources.
Provenance: The documented origin and history of a dataset: where it came from, when, how, and under what conditions.
Data integration: Combining data from multiple sources into a single coherent dataset, resolving differences in structure and meaning.

Assessment alignment

Directly assessed in Assignment 1

Assignment 1 (Data Collection Project) requires you to source, retrieve and document an original dataset. The provenance documentation you produce for that submission is a direct application of this unit.