Diagnosing a Page-Fault Storm

01 Overview

02 Learning outcomes addressed

This discussion assesses CLO4 (explain virtual-memory management), CLO6 (apply Linux systems programming and tools), and CLO8 (design and implement OS-related components).

03 Discussion prompt

04 Suggested experiments and tools

4.1 A synthetic workload

Write a program that allocates a large array and traverses it. Vary the size of the array: a size that fits comfortably in physical memory, a size that is close to the limit, and a size that exceeds physical memory. Run the program in a loop and observe the fault rate.

#include <stdlib.h>
#include <stdio.h>

int main(int argc, char **argv) {
    size_t size = (argc > 1 ? atol(argv[1]) : 1) * 1024ULL * 1024ULL * 1024ULL;
    char *buf = malloc(size);
    if (!buf) { perror("malloc"); return 1; }
    for (size_t i = 0; i < size; i += 4096) {
        buf[i] = (char)i;     /* touch one byte per page */
    }
    printf("touched %zu bytes\n", size);
    free(buf);
    return 0;
}

Each buf[i] touches a distinct page (because of the 4096 stride). The write causes a page fault the first time each page is touched. Run the program with sizes of 1 GB, 4 GB, and 8 GB on a machine with, say, 4 GB of RAM, and compare the fault rates.

4.2 A multi-process experiment

Run N copies of the program from § 4.1 in parallel, for N = 1, 2, 4, 8. Record the total throughput (bytes touched per second, across all processes) and the total fault rate. On a machine with limited memory, the throughput will drop as N grows beyond a threshold — that is thrashing.

4.3 Tools

• /usr/bin/time -v ./program — reports the elapsed time, the CPU time (user and system), the maximum resident set size, and the minor and major page faults. This is the fastest way to get a snapshot of the workload's memory behaviour.
• /proc/<pid>/stat — fields 10 (minor faults), 12 (major faults), 14 (user time in clock ticks), 15 (system time). Poll it periodically to get a time series.
• vmstat 1 — reports the system-wide swap-in (si) and swap-out (so) rates, the number of runnable processes (r), and the CPU time breakdown (us, sy, wa, id). The wa column (I/O wait) will be high during thrashing.
• perf stat -e page-faults,minor-faults,major-faults ./program — counts faults using hardware performance counters; the counts may differ slightly from /proc because they are measured at a different point.
• /proc/vmstat — fields pgfault and pgmajfault (system-wide), and pswpin and pswpout (system-wide swap).

4.4 Notes on interpretation

• Minor faults are not necessarily a problem. Every page of every file that a program reads causes a minor fault the first time it is touched, but the page is already in memory (in the page cache), so the fault is cheap.
• Major faults are expensive: they require disk I/O. A high rate of major faults is a strong indicator of thrashing.
• The wa column in vmstat is the fraction of CPU time spent waiting for I/O. It rises during thrashing because every faulting process is blocked on the swap device.
• vm.swappiness controls how aggressively the kernel reclaims anonymous pages (as opposed to file-backed pages). A high value (the default, 60) means the kernel is willing to swap anonymous memory; a low value (10 or less) means it prefers to keep anonymous memory resident and reclaim file-backed pages instead. On a system with a large working set and little swap, lowering swappiness can help; on a system with plenty of swap, the default is usually fine.

05 Grading rubric

The discussion is marked out of 100 points. The criteria and their weights are given below.

5.1 Post quality (40 points)

5.2 Peer response quality (30 points)

The descriptors are the same as for Discussion 3. An exemplary reply might:

• Suggest a different tuning that the classmate did not consider and predict how it would compare.
• Point out a metric the classmate should have measured (for example, the working-set size, or the TLB miss rate).
• Compare the classmate's results with their own and explain the difference in terms of hardware or workload.

5.3 Engagement and timeliness (20 points)

The descriptors are the same as for Discussion 1.

5.4 Netiquette and clarity (10 points)

The descriptors are the same as for Discussion 1.

06 Resources

• Silberschatz, A., Galvin, P. B., & Gagne, G. Operating System Concepts, 10th ed., Chapter 10 (Virtual Memory).
• Arpaci-Dusseau, R. H., & Arpaci-Dusseau, A. C. Operating Systems: Three Easy Pieces, Chapters 18–23 (Paging, TLBs, Swapping, Policy).
• Tanenbaum, A. S., & Bos, H. Modern Operating Systems, 4th ed., Chapter 3 (Memory Management, Section on Virtual Memory).
• Denning, P. J. The Working Set Model for Program Behavior. Communications of the ACM, 1968.
• Belady, L. A., Nelson, R. A., & Shedler, G. S. An Anomaly in Space-Time Characteristics of Certain Programs Running in a Paging Machine. Communications of the ACM, 1969.
• Gorman, M. Understanding the Linux Virtual Memory Manager. Prentice Hall, 2004. (Available online.)
• Linux kernel documentation: Documentation/admin-guide/mm/, Documentation/admin-guide/sysctl/vm.rst.
• Linux man-pages: man 5 proc (/proc/<pid>/stat, /proc/<pid>/smaps, /proc/vmstat), man 1 vmstat, man 1 pidstat, man 1 perf-stat.
• Drepper, U. What Every Programmer Should Know About Memory. Red Hat, 2007.