Tutorial 6: File Handling – Text Files vs. Binary Files

Learning Objectives

Explain files and discuss the differences between text files and binary files and the methods and functions available for manipulating files.

Overview

Files are the primary means of persistent data storage on a computer. Python provides a rich set of tools to read from and write to files, whether they contain human‑readable text (like .txt, .csv, .json) or raw binary data (images, audio, executables, etc.). This tutorial covers everything you need to know to work with files safely and efficiently.

We will start with the open() function, explore the various file modes, and then dive deep into the differences between text and binary files. You will learn the essential methods for reading and writing, the importance of specifying character encodings, and the use of the with statement for automatic resource management. We’ll also touch on handling file paths and common pitfalls.

1. Opening Files – The `open()` Function

The built‑in open() function is your gateway to file I/O. It returns a file object (also called a file handle) that provides methods for reading, writing, and closing the file.

file = open('example.txt', 'r')   # open for reading (default mode)
# ... do something with file ...
file.close()                      # always close when done!

However, relying on manual .close() is error‑prone – exceptions can leave the file open. The context manager with is the recommended way.

1.1 File Modes – A Detailed Look

The second argument to open() is the mode, a string that specifies how the file will be used. The most common are:

Mode	Description
`'r'`	Read (default). Opens the file for reading; raises `FileNotFoundError` if the file does not exist.
`'w'`	Write. Opens for writing; overwrites the file if it exists, or creates a new one if it doesn’t.
`'a'`	Append. Opens for writing; data is appended to the end of the file; creates the file if it doesn’t exist.
`'x'`	Exclusive creation. Opens for writing, but fails with `FileExistsError` if the file already exists.
`'r+'`	Read and write (must exist).
`'w+'`	Write and read (overwrites existing).
`'a+'`	Append and read (append at end).

Additionally, you can combine modes with the following modifiers:

'b' – binary mode (e.g., 'rb', 'wb').
't' – text mode (default, e.g., 'rt' is same as 'r').

1.2 Specifying Encoding for Text Files

When you open a file in text mode (the default), Python decodes the bytes into strings using a specific character encoding. The default encoding is platform‑dependent (locale.getpreferredencoding()), which may cause compatibility issues. It is best practice to explicitly specify an encoding, usually 'utf-8'.

with open('example.txt', 'r', encoding='utf-8') as f:
    content = f.read()

If you try to read a file with the wrong encoding, you will encounter a UnicodeDecodeError. We’ll discuss handling encoding errors later.

1.3 The `with` Statement – Safe File Handling

The with statement (a context manager) ensures that the file is properly closed after the block is exited, even if an exception occurs.

with open('data.txt', 'w', encoding='utf-8') as f:
    f.write('Hello, world!')
# File is automatically closed here.

You can also open multiple files in one with:

with open('source.txt', 'r') as src, open('dest.txt', 'w') as dst:
    dst.write(src.read())

2. Text Files vs. Binary Files – The Core Difference

2.1 Text Files

Representation: Store data as a sequence of characters encoded in a specific character set (e.g., UTF‑8, ASCII, Latin‑1).
Human‑readable: You can open them in a text editor and read the content.
Line endings: Python automatically translates newline characters between platform‑specific conventions (e.g., \r\n on Windows, \n on Unix) when in text mode.
Common extensions: .txt, .csv, .json, .xml, .html, .py.
Operations: Reading returns strings; writing accepts strings.
Encoding is crucial – you must specify the correct one.

2.2 Binary Files

Representation: Store data as a sequence of raw bytes (0‑255 values).
Not human‑readable: They may contain non‑textual data like images, audio, compressed files, or executables.
No encoding: There is no character decoding; you work with bytes objects.
Line endings: No translation – bytes are read/written exactly as they are.
Common extensions: .png, .jpg, .mp3, .pdf, .exe, .pyc.
Operations: Reading returns bytes; writing accepts bytes (or byte‑like objects).

2.3 When to Use Which?

Use text mode for configuration files, logs, user‑generated content, data interchange (JSON, CSV), and any file you intend to read as text.
Use binary mode for media files, serialized data (pickle), compressed archives, and any file that is not meant to be interpreted as text.

3. Essential File Methods

3.1 Reading Methods

Method	Description
`read(size=-1)`	Reads `size` characters (text) or bytes (binary). If `size` is omitted or negative, reads the entire file.
`readline(size=-1)`	Reads one line up to `size` characters/bytes. Returns an empty string when EOF is reached.
`readlines(hint=-1)`	Reads all lines and returns a list of strings (text) or bytes (binary). `hint` can limit the total number of lines read.

Example – text file:

with open('poem.txt', 'r', encoding='utf-8') as f:
    for line in f:          # iterating over the file object reads line by line
        print(line.strip())

Example – binary file:

with open('image.png', 'rb') as f:
    data = f.read(1024)     # read first 1024 bytes
    while data:
        # process data...
        data = f.read(1024)

3.2 Writing Methods

Method	Description
`write(s)`	Writes the string `s` (text mode) or bytes‑like object `s` (binary mode). Returns the number of characters/bytes written.
`writelines(lines)`	Writes a list (or any iterable) of strings/bytes to the file. Does not automatically add line breaks – you must include them.

Example:

lines = ["First line\n", "Second line\n"]
with open('output.txt', 'w', encoding='utf-8') as f:
    f.writelines(lines)

3.3 Seeking and Telling

For random access, you can move the file position:

tell() – returns the current position (as an integer) from the beginning of the file.
seek(offset, whence=0) – moves the position to offset bytes/characters from whence (0 = start, 1 = current, 2 = end).

with open('data.bin', 'rb') as f:
    f.seek(10)        # move to byte 10 from start
    byte = f.read(1)  # read the 11th byte

3.4 Other Useful Methods

flush() – forces the buffer to be written to disk (usually not needed, but useful for interactive programs).
close() – closes the file; you should rarely need to call it directly if using with.
fileno() – returns the underlying file descriptor (advanced).

4. Handling File Paths

4.1 Absolute vs. Relative Paths

Absolute path: Full path from the root of the filesystem (e.g., C:\Users\Name\data.txt on Windows, /home/name/data.txt on Linux).
Relative path: Path relative to the current working directory (e.g., data.txt or ./data.txt).

You can get the current working directory with os.getcwd() and change it with os.chdir().

4.2 Using `os.path` and `pathlib`

The os.path module provides functions for manipulating paths in a platform‑independent way:

os.path.join('dir', 'file.txt') – builds the correct path separators.
os.path.exists(path) – checks if a file or directory exists.
os.path.isfile(path) – checks if it is a file.
os.path.isdir(path) – checks if it is a directory.

The newer pathlib module (Python 3.4+) offers an object‑oriented approach:

from pathlib import Path
p = Path('data/input.txt')
print(p.exists())
print(p.parent)  # data
with p.open('r', encoding='utf-8') as f:
    content = f.read()

pathlib is recommended for new code.

5. Working with Encodings – Common Pitfalls

5.1 Specifying Encoding

Always specify encoding when opening text files. For maximum compatibility, use 'utf-8' or 'utf-8-sig' (for files with a BOM – byte order mark, often from Windows tools).

# Write with explicit encoding
with open('file.txt', 'w', encoding='utf-8') as f:
    f.write('unicode ✓')

5.2 Handling Encoding Errors

When reading a file with the wrong encoding, Python raises UnicodeDecodeError. You can handle this by specifying the errors parameter:

errors='strict' – default, raises an error.
errors='ignore' – silently skips problematic characters.
errors='replace' – replaces them with a placeholder (e.g., �).
errors='backslashreplace' – replaces with escaped sequences (like \u1234).

with open('problematic.txt', 'r', encoding='ascii', errors='replace') as f:
    content = f.read()   # malformed chars become '�'

5.3 Detecting Encoding

Sometimes you don't know the encoding. The chardet library can help detect it, but it's not built‑in.

6. Common Patterns and Examples

6.1 Reading a Text File Line by Line

with open('log.txt', 'r', encoding='utf-8') as f:
    for line in f:
        if 'ERROR' in line:
            print(line.strip())

6.2 Writing a CSV‑like File

data = [['Name', 'Age'], ['Alice', 30], ['Bob', 25]]
with open('people.csv', 'w', encoding='utf-8') as f:
    for row in data:
        f.write(','.join(str(cell) for cell in row) + '\n')

6.3 Copying a Binary File (efficient chunk‑wise)

def copy_file(src, dst):
    with open(src, 'rb') as src_f, open(dst, 'wb') as dst_f:
        while True:
            chunk = src_f.read(4096)  # 4KB chunks
            if not chunk:
                break
            dst_f.write(chunk)

6.4 Appending to a Text File

with open('log.txt', 'a', encoding='utf-8') as f:
    f.write(f'Error: {error_msg}\n')

6.5 Reading the Last N Lines (Using `seek` and looping from the end) – more advanced

def tail(filename, n=10):
    with open(filename, 'rb') as f:
        f.seek(0, 2)          # go to end
        size = f.tell()
        block = 1024
        lines = []
        while len(lines) <= n and size > 0:
            f.seek(max(0, size - block), 0)
            data = f.read(block)
            lines = data.splitlines() + lines
            size -= block
        return lines[-n:]

7. Exception Handling with Files

File operations can raise exceptions:

FileNotFoundError – when opening a file for reading that doesn’t exist.
PermissionError – when you don’t have the necessary permissions.
IsADirectoryError – when you try to open a directory as a file.
UnicodeDecodeError / UnicodeEncodeError – encoding issues.

It is good practice to catch these errors:

try:
    with open('data.txt', 'r', encoding='utf-8') as f:
        content = f.read()
except FileNotFoundError:
    print("File not found.")
except PermissionError:
    print("Permission denied.")
except Exception as e:
    print(f"An error occurred: {e}")

📝 Quiz – Check Your Understanding

Which mode should you use to open a text file for reading, and you want to raise an error if the file does not exist?
- (A) 'r'
- (B) 'w'
- (C) 'a'
- (D) 'x'
Answer
(A) `'r'`
True or False: When you open a file in binary mode, you do not need to specify an encoding.

Answer
True
What is the purpose of the with statement in file handling?
- (A) To open a file in binary mode.
- (B) To automatically close the file after the block ends.
- (C) To read the file line by line.
- (D) To handle encoding errors.
Answer
(B) To automatically close the file after the block ends.
What method reads the entire contents of a file as a string (in text mode)?
- (A) readline()
- (B) read()
- (C) readlines()
- (D) write()
Answer
(B) `read()`
Which method writes a list of strings to a file without automatically adding line breaks?
- (A) write()
- (B) writelines()
- (C) append()
- (D) extend()
Answer
(B) `writelines()`
What encoding is recommended for maximum portability?
- (A) 'ascii'
- (B) 'latin-1'
- (C) 'utf-8'
- (D) 'cp1252'
Answer
(C) `'utf-8'`
What does file.tell() return?
- (A) The number of lines in the file.
- (B) The current file position from the beginning.
- (C) The size of the file in bytes.
- (D) The file name.
Answer
(B) The current file position from the beginning.
Which mode would you use to open a binary file for reading?
- (A) 'r'
- (B) 'rb'
- (C) 'rt'
- (D) 'r+'
Answer
(B) `'rb'`
What exception is raised when you try to open a non‑existent file in read mode?
- (A) FileNotFoundError
- (B) PermissionError
- (C) OSError
- (D) IOError
Answer
(A) `FileNotFoundError`
How do you append text to the end of an existing text file without erasing its content?
- (A) Use mode 'w'
- (B) Use mode 'a'
- (C) Use mode 'x'
- (D) Use mode 'r+'
Answer
(B) Use mode `'a'`

💻 Exercises – Practice Makes Perfect

Exercise 1: Greeting File
Write a program that:

Asks the user for their name and age.
Writes a greeting to a file called greeting.txt.
Then reads the file and prints its content.

Sample Solution

```python name = input("Name: ") age = input("Age: ") with open('greeting.txt', 'w', encoding='utf-8') as f: f.write(f"Hello {name}, you are {age} years old.") with open('greeting.txt', 'r', encoding='utf-8') as f: print(f.read()) ```

Exercise 2: Line Numbering
Write a script that reads a text file input.txt and writes a new file numbered.txt where each line is prefixed with its line number.

Sample Solution

```python with open('input.txt', 'r', encoding='utf-8') as infile, \ open('numbered.txt', 'w', encoding='utf-8') as outfile: for i, line in enumerate(infile, start=1): outfile.write(f"{i}: {line}") ```

Exercise 3: Binary File Copy
Write a function copy_binary(src, dst, chunk_size=1024) that copies a binary file in chunks.

Sample Solution

```python def copy_binary(src, dst, chunk_size=1024): with open(src, 'rb') as src_f, open(dst, 'wb') as dst_f: while True: chunk = src_f.read(chunk_size) if not chunk: break dst_f.write(chunk) ```

Exercise 4: Word Counter
Read a text file, count words, lines, and characters, and write statistics to stats.txt.

Sample Solution

```python with open('sample.txt', 'r', encoding='utf-8') as f: content = f.read() lines = content.splitlines() words = content.split() chars = len(content) with open('stats.txt', 'w', encoding='utf-8') as f: f.write(f"Lines: {len(lines)}\nWords: {len(words)}\nCharacters: {chars}\n") ```

Exercise 5: CSV Reader with Dict
Given a CSV file students.csv with columns Name,Math,Science,English, read it, compute averages per student and per subject, and print.

Sample Solution

```python import csv with open('students.csv', 'r', encoding='utf-8') as f: reader = csv.DictReader(f) rows = list(reader) for row in rows: avg = (int(row['Math']) + int(row['Science']) + int(row['English'])) / 3 print(f"{row['Name']}: {avg:.2f}") # Subject averages math_avg = sum(int(r['Math']) for r in rows) / len(rows) # etc. ```

🏠 Homework – Deeper Thinking

Short Answer Questions

1. Log File Analyzer
Given a log file with lines timestamp,level,message, count levels, print summary, and write ERROR lines to errors.log.

Sample Answer

```python level_counts = {'INFO':0, 'WARNING':0, 'ERROR':0} errors = [] try: with open('server.log', 'r', encoding='utf-8') as f: for line in f: parts = line.strip().split(',', 2) if len(parts) == 3: ts, level, msg = parts if level in level_counts: level_counts[level] += 1 if level == 'ERROR': errors.append(line) except FileNotFoundError: print("File not found. Please specify another file.") # Print summary and write errors with open('errors.log', 'w', encoding='utf-8') as f: f.writelines(errors) ```

2. Find and Replace in File
Write a script that reads a file, replaces all occurrences of a search string, and overwrites the file.

Sample Answer

```python filename = input("Filename: ") search = input("Search: ") replace = input("Replace: ") with open(filename, 'r', encoding='utf-8') as f: content = f.read() new_content = content.replace(search, replace) with open(filename, 'w', encoding='utf-8') as f: f.write(new_content) ```

3. Directory Tree Lister
Recursively list all files and sizes in a directory with indentation.

Sample Answer

```python from pathlib import Path def list_dir(path, indent=0): for item in Path(path).iterdir(): print(' ' * indent + item.name, end='') if item.is_file(): print(f" ({item.stat().st_size} bytes)") else: print() list_dir(item, indent+1) ```

4. Binary File Integrity Checker (Checksum)
Calculate SHA‑256 hash and verify.

Sample Answer

```python import hashlib def hash_file(filename): h = hashlib.sha256() with open(filename, 'rb') as f: for chunk in iter(lambda: f.read(4096), b''): h.update(chunk) return h.hexdigest() # Write hash to file, then verify ```

Essay Questions

5. CSV to JSON Converter
Write a function csv_to_json(csv_filename, json_filename) using csv and json modules.

Sample Answer

```python import csv, json def csv_to_json(csv_filename, json_filename): with open(csv_filename, 'r', encoding='utf-8') as csv_f: reader = csv.DictReader(csv_f) data = list(reader) with open(json_filename, 'w', encoding='utf-8') as json_f: json.dump(data, json_f, indent=2) ```

Homework Hints

Q1: Use str.startswith to identify log levels? Better: split by comma. Use a dictionary for counts.
Q2: Read entire file with read(), then replace(), then seek to beginning and write, or write to a temp file and rename.
Q3: Use pathlib.Path.rglob('*') or os.walk(). Keep track of depth for indentation.
Q4: Use hashlib.sha256() and update with file content chunks to avoid memory issues.
Q5: Use csv.DictReader – it yields dictionaries directly. Then json.dump() the list.

Summary

In this tutorial, you have learned:

How to open files with open() and the importance of the with statement.
The critical distinction between text files and binary files – encodings vs. raw bytes.
Essential methods for reading and writing, including chunked processing for large files.
How to handle file paths with os.path and pathlib.
Common pitfalls with encodings and how to manage them.
Exception handling for robust file operations.

With these skills, you can handle persistent data storage in your Python applications, from simple configuration files to complex binary data processing.

Next Steps: In Tutorial 7, we will cover Exception Handling in depth, learning how to anticipate, catch, and manage errors gracefully across your codebase.

Happy file handling!

Previous | Tutorial index | Next

Tutorial 6: File Handling – Text Files vs. Binary Files

Learning Objectives

Overview

1. Opening Files – The open() Function

1.1 File Modes – A Detailed Look

1.2 Specifying Encoding for Text Files

1.3 The with Statement – Safe File Handling

2. Text Files vs. Binary Files – The Core Difference

2.1 Text Files

2.2 Binary Files

2.3 When to Use Which?

3. Essential File Methods

3.1 Reading Methods

3.2 Writing Methods

3.3 Seeking and Telling

3.4 Other Useful Methods

4. Handling File Paths

4.1 Absolute vs. Relative Paths

4.2 Using os.path and pathlib

5. Working with Encodings – Common Pitfalls

5.1 Specifying Encoding

5.2 Handling Encoding Errors

5.3 Detecting Encoding

6. Common Patterns and Examples

6.1 Reading a Text File Line by Line

6.2 Writing a CSV‑like File

6.3 Copying a Binary File (efficient chunk‑wise)

6.4 Appending to a Text File

6.5 Reading the Last N Lines (Using seek and looping from the end) – more advanced

7. Exception Handling with Files

📝 Quiz – Check Your Understanding

💻 Exercises – Practice Makes Perfect

🏠 Homework – Deeper Thinking

Short Answer Questions

Essay Questions

Homework Hints

Summary

1. Opening Files – The `open()` Function

1.3 The `with` Statement – Safe File Handling

4.2 Using `os.path` and `pathlib`

6.5 Reading the Last N Lines (Using `seek` and looping from the end) – more advanced