Tutorial 7: Higher‑Order Functions – `map`, `filter`, and `reduce`

Learning Objectives

Use special functions such as mapping, filtering, and reducing for functional‑style data processing.

Overview

Higher‑order functions are functions that operate on other functions—either by taking them as arguments or by returning them. Python provides several built‑in higher‑order functions that are extremely useful for processing iterables. The three most important ones are map, filter, and reduce. Together with lambda expressions, they enable a concise, functional programming style for data transformation, selection, and aggregation. In this tutorial, you’ll learn how to use these functions effectively, when they are appropriate, and how they compare to more Pythonic alternatives like comprehensions.

1. What Is a Higher‑Order Function?

A higher‑order function is any function that:

Takes one or more functions as arguments, or
Returns a function as its result.

Python functions are first‑class objects, so they can be passed around like any other value. This makes higher‑order functions natural in Python.

Examples you’ve already seen:

sorted(iterable, key=func) – key is a function.
map(func, iterable) – func is applied to each element.
filter(func, iterable) – func decides which elements to keep.

2. The `map()` Function

2.1 Purpose and Syntax

map(function, iterable, ...) applies function to every element of the iterable(s) and returns an iterator that yields the results. The number of iterables passed must match the number of arguments expected by function.

map(func, iterable1, iterable2, ...)

2.2 Basic Examples

# Square numbers
numbers = [1, 2, 3, 4]
squared = map(lambda x: x ** 2, numbers)
print(list(squared))   # [1, 4, 9, 16]

Important: map returns an iterator, not a list. You must convert it to a list (or iterate over it) to see the results. This is memory‑efficient for large data.

2.3 Using `map` with Multiple Iterables

a = [1, 2, 3]
b = [10, 20, 30]
sums = map(lambda x, y: x + y, a, b)
print(list(sums))   # [11, 22, 33]

The function receives one element from each iterable in parallel. If the iterables are of different lengths, map stops when the shortest is exhausted.

2.4 `map` vs. List Comprehension

List comprehensions are often more readable and Pythonic than map with a lambda:

# Using map
squared = list(map(lambda x: x**2, numbers))
# Using comprehension
squared = [x**2 for x in numbers]

For simple transformations, comprehensions are preferred. Use map when you already have a pre‑defined function and don't want to rewrite it inside a comprehension.

3. The `filter()` Function

3.1 Purpose and Syntax

filter(function, iterable) constructs an iterator from elements of the iterable for which function returns True. If function is None, it filters out falsy elements (like None, False, 0, empty strings, etc.).

filter(func, iterable)

3.2 Basic Example

numbers = [1, 2, 3, 4, 5, 6]
evens = filter(lambda x: x % 2 == 0, numbers)
print(list(evens))   # [2, 4, 6]

3.3 Using `filter` with `None`

mixed = [0, 1, False, True, None, "hello", ""]
truthy = filter(None, mixed)
print(list(truthy))   # [1, True, "hello"]

3.4 `filter` vs. Comprehension

Again, comprehensions are often clearer:

evens = [x for x in numbers if x % 2 == 0]

4. The `reduce()` Function (from `functools`)

4.1 Purpose and Syntax

reduce(function, iterable[, initializer]) applies function cumulatively to the items of iterable, reducing the iterable to a single value. It is not built‑in; it must be imported from functools.

from functools import reduce
reduce(func, iterable, initializer=None)

4.2 Example – Product of Elements

from functools import reduce
numbers = [1, 2, 3, 4]
product = reduce(lambda x, y: x * y, numbers)
print(product)   # 24

The reduction proceeds as: (((1*2)*3)*4) = 24.

4.3 Using an Initializer

If you provide an initializer, it is placed before the iterable items in the reduction.

product = reduce(lambda x, y: x * y, numbers, 10)
print(product)   # 240  (10 * 1 * 2 * 3 * 4)

If the iterable is empty, reduce returns the initializer; without an initializer, it raises TypeError.

4.4 Common Operations with `reduce`

Sum (though sum() is built‑in for numbers).
Maximum (though max() exists).
Concatenating strings.
Flattening lists.

4.5 When to Use `reduce`

reduce is less common than map and filter in modern Python because explicit loops or sum, any, all, etc., are often clearer. However, it shines in functional‑style pipelines.

5. Practical Data Processing Pipelines

Higher‑order functions can be chained to create data pipelines:

data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Pipeline: square each, keep evens, sum
result = reduce(
    lambda acc, x: acc + x,
    filter(
        lambda x: x % 2 == 0,
        map(lambda x: x ** 2, data)
    ),
    0
)
print(result)   # 220  (4 + 16 + 36 + 64 + 100)

This is more readable if broken into steps:

squared = map(lambda x: x**2, data)
even_squared = filter(lambda x: x % 2 == 0, squared)
total = reduce(lambda a, b: a + b, even_squared, 0)

6. Caveats and Best Practices

6.1 Lazy Evaluation

map and filter return iterators that evaluate lazily. This is efficient but can lead to confusion if you try to reuse them—they are exhausted after one pass.

m = map(lambda x: x*2, [1,2,3])
print(list(m))   # [2,4,6]
print(list(m))   # []  (iterator is exhausted)

If you need to reuse, convert to a list.

6.2 Side Effects in `map` and `filter`

These functions are meant for pure functions. Using them with functions that have side effects (like print) is considered bad practice. Use explicit loops instead.

6.3 Pre‑existing Functions

If you already have a function defined, you can pass it directly:

def square(x): return x ** 2
squared = list(map(square, numbers))

6.4 Pythonic Alternatives

List comprehensions and generator expressions are often more readable and faster than map/filter with lambdas:

[f(x) for x in iterable] instead of map(f, iterable)
[x for x in iterable if pred(x)] instead of filter(pred, iterable)
Use built‑in sum, any, all, max, min instead of reduce when possible.

However, map and filter are still useful when you need to apply an existing function to multiple iterables (e.g., map(operator.add, a, b)) or when you’re working with functional patterns.

7. Advanced: `map` with Multiple Iterables and `zip`

map can be used to combine values from multiple sequences:

list1 = [1, 2, 3]
list2 = [10, 20, 30]
list3 = [100, 200, 300]
sums = list(map(lambda a, b, c: a + b + c, list1, list2, list3))  # [111, 222, 333]

This is similar to using zip with a comprehension.

📝 Quiz – Check Your Understanding

What does map return?
- (A) A list
- (B) A tuple
- (C) An iterator
- (D) A dictionary
Answer
(C) An iterator
Which function from functools is used for cumulative reduction?
- (A) accumulate
- (B) reduce
- (C) fold
- (D) aggregate
Answer
(B) `reduce`
Given numbers = [1, 2, 3, 4], what is the result of list(filter(lambda x: x > 2, numbers))?
- (A) [3, 4]
- (B) [1, 2]
- (C) [2, 3, 4]
- (D) [1, 2, 3, 4]
Answer
(A) `[3, 4]`
True or False: reduce is a built‑in function in Python and does not need to be imported.

Answer
False – it’s in `functools`.
What is the output of list(map(lambda x: x * 2, [1, 2, 3]))?
- (A) [2, 4, 6]
- (B) [1, 2, 3]
- (C) [1, 4, 9]
- (D) None
Answer
(A) `[2, 4, 6]`
Which of the following is more Pythonic for filtering even numbers?
- (A) filter(lambda x: x%2==0, numbers)
- (B) [x for x in numbers if x%2==0]
- (C) map(lambda x: x%2==0, numbers)
- (D) Both A and B are fine; B is often preferred for clarity.
Answer
(D) Both A and B are fine; B is often preferred.
If you call map with two iterables of different lengths, what happens?
- (A) It raises a ValueError.
- (B) It stops when the shorter iterable ends.
- (C) It pads the shorter one with None.
- (D) It repeats the shorter one to match the longer.
Answer
(B) It stops when the shorter iterable ends.
What does reduce(lambda x, y: x + y, [1, 2, 3], 10) return?
- (A) 6
- (B) 16
- (C) 10
- (D) 7
Answer
(B) `16`
Which of the following is a higher‑order function?
- (A) len()
- (B) sorted()
- (C) print()
- (D) type()
Answer
(B) `sorted()`
What is the result of list(filter(None, [0, 1, False, 2, '']))?
- (A) [0, 1, 2]
- (B) [1, 2]
- (C) [0, 1, False, 2, '']
- (D) [1, 2, '']
Answer
(B) `[1, 2]`

💻 Exercises – Practice Makes Perfect

Exercise 1: Map with Multiple Lists
Given list_a = [1, 2, 3] and list_b = [4, 5, 6], use map to compute element‑wise sums.

Sample Solution

```python list_a = [1, 2, 3] list_b = [4, 5, 6] sums = list(map(lambda a, b: a + b, list_a, list_b)) print(sums) # [5, 7, 9] ```

Exercise 2: Filter Strings
Given words = ["apple", "banana", "cherry", "date", "elderberry"], use filter to keep only words longer than 5 characters.

Sample Solution

```python words = ["apple", "banana", "cherry", "date", "elderberry"] long_words = list(filter(lambda w: len(w) > 5, words)) print(long_words) # ['banana', 'cherry', 'elderberry'] ```

Exercise 3: Reduce Product
Use reduce to compute the product of all numbers in [2, 3, 4, 5].

Sample Solution

```python from functools import reduce product = reduce(lambda x, y: x * y, [2, 3, 4, 5]) print(product) # 120 ```

Exercise 4: Chaining Pipelines
Write a pipeline that squares numbers 1–10, keeps those > 20, and sums them.

Sample Solution

```python from functools import reduce data = range(1, 11) result = reduce(lambda acc, x: acc + x, filter(lambda x: x > 20, map(lambda x: x**2, data)), 0) print(result) # 355 ```

Exercise 5: Using map with Built‑in Functions
Use map with str.upper to convert a list of lowercase strings to uppercase.

Sample Solution

```python words = ["hello", "world"] uppercase = list(map(str.upper, words)) print(uppercase) # ['HELLO', 'WORLD'] ```

🏠 Homework – Deeper Thinking

Short Answer Questions

1. Data Cleaning Pipeline
Given raw = [" 12 ", "34.5", " -7 ", "abc", " 0 "], write a pipeline that strips, converts to float (discarding invalid), and sums.

Sample Answer

```python def to_float(s): try: return float(s.strip()) except ValueError: return None

raw = [" 12 ", "34.5", " -7 ", "abc", " 0 "] valid = filter(lambda v: v is not None, map(to_float, raw)) total = sum(valid) print(total) # 39.5

</details>

**2. Custom `map` Implementation**  
Write `my_map(func, *iterables)` that mimics `map` using `zip` and `yield`.  

<details><summary>Sample Answer</summary>
```python
def my_map(func, *iterables):
    for args in zip(*iterables):
        yield func(*args)

print(list(my_map(lambda x, y: x + y, [1,2,3], [4,5,6])))   # [5,7,9]

3. Grouping and Aggregation with reduce
Use reduce to count numbers by tens digit (0–9).

Sample Answer

```python from functools import reduce numbers = [5, 12, 23, 8, 31] counts = reduce(lambda acc, x: acc.update({x//10: acc.get(x//10,0)+1}) or acc, numbers, {}) print(counts) # {0: 2, 1: 1, 2: 1, 3: 1} ```

Essay Questions

4. Performance Comparison
Compare map vs list comprehension for squaring 10 million numbers using timeit.

Sample Answer

```python import timeit setup = "numbers = range(10000000)" map_time = timeit.timeit("list(map(lambda x: x**2, numbers))", setup, number=1) comp_time = timeit.timeit("[x**2 for x in numbers]", setup, number=1) print(f"map: {map_time:.4f}s, comprehension: {comp_time:.4f}s") ```

5. Functional JSON Transformation
Given people, filter age ≥ 30, extract names, join with commas.

Sample Answer

```python people = [{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}, {"name": "Charlie", "age": 35}] result = ", ".join(map(lambda p: p['name'], filter(lambda p: p['age'] >= 30, people))) print(result) # Alice, Charlie ```

Homework Hints

Q1: Write a helper to_float(s) that returns float(s.strip()) or None. Then filter(lambda v: v is not None, map(to_float, raw)) and sum.
Q2: Use zip(*iterables) and loop to yield func(*args).
Q3: reduce(lambda acc, x: acc.update({x//10: acc.get(x//10,0)+1}) or acc, numbers, {}).
Q4: Use timeit.timeit('list(map(lambda x: x**2, range(10000000)))', number=1) vs comprehension.
Q5: Use filter(lambda p: p['age'] >= 30, people), then map(lambda p: p['name'], ...), then reduce(lambda acc, name: acc + (', ' if acc else '') + name, ..., '') or simply ', '.join(...).

Summary

In this tutorial, you have learned:

The concept of higher‑order functions.
map – applies a function to all elements of an iterable and returns an iterator.
filter – selects elements that satisfy a predicate.
reduce – cumulatively combines elements into a single value.
How to use them with lambdas and existing functions.
Their laziness and potential pitfalls.
The more Pythonic alternatives: comprehensions and built‑in aggregators.

These tools are valuable for writing expressive, functional‑style code. However, always prioritise readability and maintainability. In many cases, list comprehensions and generator expressions are clearer, but map and filter still have their niche, especially when you need to apply an existing function or work with multiple iterables.

Next Steps: In Tutorial 8, we will explore Decorators – a powerful way to modify or enhance functions without changing their source code.

Happy mapping, filtering, and reducing!

Previous | Tutorial index | Next

Tutorial 7: Higher‑Order Functions – map, filter, and reduce

Learning Objectives

Overview

1. What Is a Higher‑Order Function?

2. The map() Function

2.1 Purpose and Syntax

2.2 Basic Examples

2.3 Using map with Multiple Iterables

2.4 map vs. List Comprehension

3. The filter() Function

3.1 Purpose and Syntax

3.2 Basic Example

3.3 Using filter with None

3.4 filter vs. Comprehension

4. The reduce() Function (from functools)

4.1 Purpose and Syntax

4.2 Example – Product of Elements

4.3 Using an Initializer

4.4 Common Operations with reduce

4.5 When to Use reduce

5. Practical Data Processing Pipelines

6. Caveats and Best Practices

6.1 Lazy Evaluation

6.2 Side Effects in map and filter

6.3 Pre‑existing Functions

6.4 Pythonic Alternatives

7. Advanced: map with Multiple Iterables and zip

📝 Quiz – Check Your Understanding

💻 Exercises – Practice Makes Perfect

🏠 Homework – Deeper Thinking

Short Answer Questions

Essay Questions

Homework Hints

Summary

Tutorial 7: Higher‑Order Functions – `map`, `filter`, and `reduce`

2. The `map()` Function

2.3 Using `map` with Multiple Iterables

2.4 `map` vs. List Comprehension

3. The `filter()` Function

3.3 Using `filter` with `None`

3.4 `filter` vs. Comprehension

4. The `reduce()` Function (from `functools`)

4.4 Common Operations with `reduce`

4.5 When to Use `reduce`

6.2 Side Effects in `map` and `filter`

7. Advanced: `map` with Multiple Iterables and `zip`