Tutorial 8: Generators – `yield` and Lazy Evaluation

Learning Objectives

Explain what generators are and what advantages they have.
Define a function as a generator using the yield keyword.

Overview

Generators are a special kind of function that return values lazily—they produce one value at a time, pausing between yields, rather than computing all values at once and returning a collection. This makes them extremely memory‑efficient and ideal for working with large datasets, streaming data, or infinite sequences. In this tutorial, you’ll learn how to define generators using yield, how they maintain state, their advantages over regular functions, and how to use generator expressions for concise lazy code.

1. What Is a Generator?

A generator is a function that contains one or more yield expressions. When called, it returns a generator iterator (often just called a generator) that can be iterated over. Each time you request the next value (e.g., with next() or in a for loop), the generator executes until it hits a yield, returns the yielded value, and suspends its state—local variables, instruction pointer, etc.—until the next request.

1.1 The `yield` Keyword

yield is similar to return, but instead of terminating the function, it pauses it and remembers where it left off. When the generator is resumed, it continues from that point.

def simple_generator():
    yield 1
    yield 2
    yield 3

gen = simple_generator()
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3
# next(gen)       # StopIteration

2. How Generators Work – State Suspension

When a generator function is called, its code is not executed immediately; instead, a generator object is returned. Each call to next() (or iteration) runs the code until the next yield. Local variables and the execution state are saved between calls.

2.1 The Execution Flow

def count_down(n):
    print("Starting countdown")
    while n > 0:
        yield n
        n -= 1
    print("Done!")

c = count_down(3)
print(next(c))  # Starting countdown, then 3
print(next(c))  # 2
print(next(c))  # 1
print(next(c))  # Done! then raises StopIteration

The generator runs until it either yields or returns. When the function completes (or hits a return), it raises StopIteration.

2.2 Generator vs. Regular Function

A regular function that returns a list computes all values upfront:

def squares_list(n):
    return [i*i for i in range(n)]

This stores the whole list in memory. A generator computes values on‑the‑fly:

def squares_gen(n):
    for i in range(n):
        yield i*i

The generator uses O(1) memory, independent of n.

3. Lazy Evaluation – The Key Advantage

Lazy evaluation means that values are computed only when they are needed. This provides:

Memory efficiency: no need to store all results; ideal for large or infinite data.
Time efficiency: you might not need to process all items; you can stop early.
Composability: generators can be chained to form data pipelines without intermediate storage.

3.1 Example – Processing a Large File

Reading a huge file line by line without loading it all:

def read_large_file(file_path):
    with open(file_path, 'r') as f:
        for line in f:
            yield line

# Processing only a few lines
for line in read_large_file('huge.log'):
    if 'ERROR' in line:
        print(line)
        break   # stops early, never reads the rest

3.2 Infinite Sequences

Generators can produce infinite values:

def infinite_numbers():
    n = 0
    while True:
        yield n
        n += 1

# Take first 10
for num, _ in zip(infinite_numbers(), range(10)):
    print(num)

4. Generator Methods – `send()`, `throw()`, `close()`

Generators are bidirectional; they can receive data from the caller.

4.1 `send(value)`

Allows you to send a value back into the generator, which becomes the result of the yield expression. This enables coroutines.

def accumulator():
    total = 0
    while True:
        x = yield total
        if x is None:
            break
        total += x

acc = accumulator()
next(acc)           # prime the generator (advance to first yield)
print(acc.send(5))  # total becomes 5, yields 5
print(acc.send(3))  # total becomes 8, yields 8
acc.close()

send() requires the generator to be started (usually with next() or send(None)).

4.2 `throw(type[, value[, traceback]])`

Raises an exception inside the generator at the current yield point. This can be caught within the generator.

4.3 `close()`

Stops the generator by raising GeneratorExit inside it. Used to free resources.

5. Generator Expressions – Lazy Comprehensions

Generator expressions are a concise way to create generators with a syntax similar to list comprehensions, but using parentheses () instead of [].

# List comprehension: eager, creates full list
squares_list = [x*x for x in range(1000000)]   # memory heavy

# Generator expression: lazy, returns a generator
squares_gen = (x*x for x in range(1000000))    # memory efficient

# Use it in a loop
for sq in squares_gen:
    if sq > 1000:
        break

They are ideal for large datasets where you don't need all values at once.

6. Common Use Cases for Generators

Processing large files: lines, records, chunks.
Streaming data: network packets, sensor readings.
Infinite sequences: Fibonacci, prime numbers, random streams.
Data pipelines: chain generators to transform data step by step.
Custom iteration: when the logic for iteration is complex, a generator can encapsulate it.
Recursive traversal: e.g., walking a directory tree, generating all files.

7. Performance and Pitfalls

Generator overhead: each yield and next call has some overhead, so for small datasets, a list might be faster. For large data, memory savings often outweigh the cost.
Single‑use: generators are exhausted after one pass; you cannot reset them. To reuse, you need to recreate the generator.
Close resources: if a generator holds resources (e.g., open files), it's good practice to use try/finally or with to ensure cleanup.

def read_file_safe(filename):
    with open(filename) as f:
        for line in f:
            yield line
# The file is closed when the generator is garbage collected or exhausted.

8. Generators vs. Iterators

All generators are iterators (they implement the iterator protocol), but not all iterators are generators. A generator is created by a function with yield; a custom iterator is a class with __iter__ and __next__. Generators are simpler to write.

📝 Quiz – Check Your Understanding

Which keyword is used to define a generator?
- (A) return
- (B) yield
- (C) generator
- (D) next
Answer
(B) `yield`
What does a generator function return when called?
- (A) A list of values.
- (B) A generator object.
- (C) The first yielded value.
- (D) None.
Answer
(B) A generator object.
What happens when a generator is exhausted?
- (A) It returns None.
- (B) It raises StopIteration.
- (C) It raises GeneratorExit.
- (D) It restarts automatically.
Answer
(B) It raises `StopIteration`.
True or False: A generator can produce an infinite sequence of values.

Answer
True
What is the main advantage of using a generator over a list comprehension for large data?
- (A) Faster execution.
- (B) Lower memory consumption.
- (C) Easier syntax.
- (D) Supports indexing.
Answer
(B) Lower memory consumption.
How can you manually get the next value from a generator?
- (A) generator.next()
- (B) next(generator)
- (C) generator.__next__()
- (D) Both B and C.
Answer
(D) Both B and C.
What is a generator expression?
- (A) A list comprehension using [].
- (B) A lazy version of a comprehension using ().
- (C) A dictionary comprehension using {}.
- (D) A way to create a generator using def and yield.
Answer
(B) A lazy version of a comprehension using `()`.
Given gen = (x for x in range(3)), what does list(gen) return?
- (A) [0, 1, 2]
- (B) [0, 1, 2, 3]
- (C) [0, 1]
- (D) A generator object.
Answer
(A) `[0, 1, 2]`
What method allows you to send a value back into a generator?
- (A) send()
- (B) throw()
- (C) yield()
- (D) return()
Answer
(A) `send()`
If you call next(gen) on a generator that has just been created, where does execution start?
- (A) At the first line of the function.
- (B) At the first yield.
- (C) At the last yield.
- (D) It raises StopIteration immediately.
Answer
(B) At the first `yield`.

💻 Exercises – Practice Makes Perfect

Exercise 1: Simple Generator
Write a generator even_numbers(n) that yields even numbers from 0 up to (but not including) n.

Sample Solution

```python def even_numbers(n): for i in range(0, n, 2): yield i

for num in even_numbers(10): print(num) # 0,2,4,6,8

</details>

**Exercise 2: Fibonacci Generator**  
Write a generator `fibonacci()` that yields Fibonacci numbers indefinitely.  

<details><summary>Sample Solution</summary>
```python
def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib = fibonacci()
for _ in range(10):
    print(next(fib))   # 0,1,1,2,3,5,8,13,21,34

Exercise 3: File Reader Generator
Write a generator read_file_lines(filename) that yields each line with leading/trailing whitespace removed.

Sample Solution

```python def read_file_lines(filename): with open(filename, 'r', encoding='utf-8') as f: for line in f: stripped = line.strip() if stripped: yield stripped

Usage

for line in read_file_lines('sample.txt'): print(line)

</details>

**Exercise 4: Generator Expression**  
Given a list of numbers, use a generator expression to produce squares only for even numbers.  

<details><summary>Sample Solution</summary>
```python
numbers = [1, 2, 3, 4, 5, 6]
even_squares = (x*x for x in numbers if x % 2 == 0)
for sq in even_squares:
    print(sq)   # 4, 16, 36

Exercise 5: Chaining Generators
Write filter_positive and square_all generators, then chain them.

Sample Solution

```python def filter_positive(numbers): for n in numbers: if n > 0: yield n

def square_all(numbers): for n in numbers: yield n * n

data = [-3, -2, -1, 0, 1, 2, 3] pipeline = square_all(filter_positive(data)) print(list(pipeline)) # [1, 4, 9]

</details>

---

### 🏠 Homework – Deeper Thinking

#### Short Answer Questions

**1. Prime Number Generator**  
Write a generator `primes()` that yields prime numbers indefinitely. Use it to find the 100th prime.  

<details><summary>Sample Answer</summary>
```python
def primes():
    yield 2
    n = 3
    while True:
        is_prime = True
        for p in range(2, int(n**0.5)+1):
            if n % p == 0:
                is_prime = False
                break
        if is_prime:
            yield n
        n += 2

p = primes()
for _ in range(99):
    next(p)
print(next(p))   # 100th prime

2. Paginated API Data
Simulate a paginated API with get_page(page_num) and write a generator fetch_all() that yields all items.

Sample Answer

```python def get_page(page_num): # Simulate pages: page 0 = [0,1,2], page 1 = [3,4,5], page 2 = [] if page_num == 0: return [0,1,2] elif page_num == 1: return [3,4,5] else: return []

def fetch_all(): page = 0 while True: items = get_page(page) if not items: break for item in items: yield item page += 1

print(list(fetch_all())) # [0,1,2,3,4,5]

</details>

**3. Generator with `send()`**  
Write a generator `running_average()` that accepts numbers via `send()` and yields the running average.  

<details><summary>Sample Answer</summary>
```python
def running_average():
    total = 0
    count = 0
    while True:
        n = yield total / count if count else 0
        total += n
        count += 1

avg = running_average()
next(avg)
print(avg.send(10))   # 10.0
print(avg.send(20))   # 15.0
print(avg.send(30))   # 20.0

Essay Questions

4. Flatten Nested Iterables Recursively with Generator
Write a recursive generator flatten(iterable) that yields all elements from a deeply nested list.

Sample Answer

```python def flatten(iterable): for item in iterable: if isinstance(item, list): yield from flatten(item) else: yield item

print(list(flatten([1, [2, [3, 4]], 5]))) # [1,2,3,4,5]

</details>

**5. Data Pipeline with Generators**  
Create a pipeline of generators that reads a log file, filters for "ERROR", extracts timestamp/message, and yields tuples.  

<details><summary>Sample Answer</summary>
```python
def read_lines(filename):
    with open(filename, 'r') as f:
        for line in f:
            yield line

def filter_errors(lines):
    for line in lines:
        if 'ERROR' in line:
            yield line

def parse_lines(lines):
    for line in lines:
        # Assume format: [timestamp] message
        parts = line.strip().split('] ', 1)
        if len(parts) == 2:
            timestamp = parts[0].lstrip('[')
            message = parts[1]
            yield timestamp, message

pipeline = parse_lines(filter_errors(read_lines('server.log')))
for i, (ts, msg) in enumerate(pipeline):
    if i >= 10:
        break
    print(f"{ts}: {msg}")

Homework Hints

Q1: Use trial division up to sqrt. primes() yields prime numbers.
Q2: Simulate get_page: maintain a page counter; return list or empty. Generator: while True: page = get_page(page_num); if not page: break; for item in page: yield item; page_num += 1.
Q3:

def running_average():
    total = 0
    count = 0
    while True:
        n = yield total / count if count else 0
        total += n
        count += 1

Q4: Recursively yield: loop over iterable, if element is iterable (except strings), yield from flatten(element) else yield element.
Q5: Define generators step by step; use for to chain.

Summary

In this tutorial, you have learned:

What generators are and how to create them with yield.
The concept of lazy evaluation and its memory advantages.
How generators maintain state between calls.
Advanced methods: send(), throw(), close().
Generator expressions for concise lazy generation.
Common use cases: large files, infinite sequences, pipelines.
Pitfalls and best practices.

Generators are a powerful tool for writing efficient, memory‑friendly Python code. They are especially useful when working with big data, streaming, or building composable processing pipelines.

Next Steps: In Tutorial 9, we will explore Decorators – a way to modify or enhance functions without changing their code.

Happy generating!

Previous | Tutorial index | Next

Tutorial 8: Generators – yield and Lazy Evaluation

Learning Objectives

Overview

1. What Is a Generator?

1.1 The yield Keyword

2. How Generators Work – State Suspension

2.1 The Execution Flow

2.2 Generator vs. Regular Function

3. Lazy Evaluation – The Key Advantage

3.1 Example – Processing a Large File

3.2 Infinite Sequences

4. Generator Methods – send(), throw(), close()

4.1 send(value)

4.2 throw(type[, value[, traceback]])

4.3 close()

5. Generator Expressions – Lazy Comprehensions

6. Common Use Cases for Generators

7. Performance and Pitfalls

8. Generators vs. Iterators

📝 Quiz – Check Your Understanding

💻 Exercises – Practice Makes Perfect

Usage

Essay Questions

Homework Hints

Summary

Tutorial 8: Generators – `yield` and Lazy Evaluation

1.1 The `yield` Keyword

4. Generator Methods – `send()`, `throw()`, `close()`

4.1 `send(value)`

4.2 `throw(type[, value[, traceback]])`

4.3 `close()`