📈 Tutorial 10: TCP Performance Analysis and Advanced Features
University‑level treatment – COMP347 (TrustOpen University)
🎯 Learning Objectives
After completing this tutorial, you should be able to:
- Analyze TCP throughput using window‑based transmission models.
- Calculate the bandwidth‑delay product and its impact on link utilization.
- Explain the effects of RTT on TCP performance and fairness.
- Describe advanced TCP features: SACK, Window Scaling, Timestamps, and their benefits.
- Evaluate the role of Nagle's algorithm and delayed ACKs in performance.
- Compare modern TCP variants (Cubic, BBR) with classic Reno.
- Discuss performance trade‑offs and tuning parameters.
🔍 Overview
TCP performance is governed by a complex interplay of window sizes, round‑trip times, loss rates, and receiver constraints. This tutorial provides a deep dive into TCP throughput analysis, the bandwidth‑delay product, and the impact of RTT and loss. We then explore advanced features that enhance TCP’s performance in modern networks: Selective Acknowledgment (SACK), Window Scaling, Timestamps, Nagle's algorithm, and Delayed ACKs. Finally, we examine modern congestion control algorithms like Cubic and BBR, and discuss performance tuning and real‑world case studies, including high‑speed networks and data centers.
📘 1. Introduction to TCP Performance
TCP performance is determined by the sender’s ability to keep the network pipe full while avoiding congestion. The key factors are:
- Window size (effective window = min(cwnd, rwnd)).
- Round‑trip time (RTT).
- Packet loss rate and recovery mechanisms.
- Receiver buffer and advertised window.
- Network path characteristics (bandwidth, delay, queuing).
The fundamental relationship is: Throughput ≤ window / RTT, with the equality holding when the window is fully utilized.
📘 2. Throughput Analysis: Window‑Based Transmission
For a connection with a constant window size W (in bytes) and RTT, the maximum throughput is:
Throughput = W / RTT
However, if the window is limited by the receiver (rwnd) or by congestion (cwnd), the effective window is the minimum. Also, if the link bandwidth is B, the throughput cannot exceed B.
In practice, TCP’s window varies over time (sawtooth). The average throughput under steady state with loss rate p is approximated by the square‑root law (covered in Tutorial 9), but for large windows, the simple W/RTT model gives a reasonable upper bound.
We can also express throughput in terms of the sending rate: rate = (cwnd / RTT) * MSS.
📘 3. Bandwidth‑Delay Product and Its Implications
The bandwidth‑delay product (BDP) is the amount of data that can be in transit in the network: BDP = bandwidth × RTT. To fully utilize the link, the window size must be at least BDP.
- If
window < BDP, the link is underutilized (the sender cannot fill the pipe).
- If
window ≥ BDP, the link can be fully utilized.
- Large BDPs require large windows, which may exceed the 16‑bit window field (65535 bytes) unless window scaling is used.
Example: A 1 Gbps link with RTT = 100 ms has BDP = 1e9 * 0.1 = 100 Mbits = 12.5 MB. To saturate this link, the window must be at least 12.5 MB. Without window scaling, the maximum window is 64 KB, so utilization would be only 64KB/12.5MB ≈ 0.5%.
📘 4. RTT Effects and Performance
RTT affects TCP in several ways:
- Throughput: For a given window, throughput = window / RTT. Longer RTTs reduce throughput.
- Fairness: Flows with shorter RTTs get more bandwidth because they increase cwnd faster (ACK clocking).
- Timeout estimation: RTT variability affects RTO, influencing loss recovery time.
TCP’s congestion control uses ACK clocking: each ACK allows a new segment to be sent, effectively pacing the sender at the rate of ACK arrival. If the ACK arrival rate is slow (high RTT), the sender’s rate is limited.
📘 5. TCP Sender and Receiver Constraints
- Sender: Limited by cwnd (congestion) and rwnd (flow control). Also limited by the send buffer size (SO_SNDBUF).
- Receiver: Advertises rwnd based on available buffer space. Also limited by the receive buffer size (SO_RCVBUF).
- Application: The rate at which the application reads/writes data can affect performance.
If the application is slow to read data, the receiver window will shrink, causing the sender to slow down. Similarly, if the sender application writes data slowly, the connection may be idle.
📘 6. Advanced Features: SACK, Window Scaling, Timestamps
- Selective Acknowledgment (SACK): Allows the receiver to acknowledge non‑contiguous blocks of data, enabling the sender to retransmit only the lost segments. Improves recovery from multiple losses.
- Window Scaling: Negotiates a shift value (0–14) to scale the advertised window, allowing windows up to 2^30 bytes (1 GB). Essential for high BDP networks.
- Timestamps: Provide two 32‑bit timestamp fields (TSval, TSecr) used for accurate RTT measurement and Protection Against Wrapped Sequence numbers (PAWS).
These features are negotiated during the three‑way handshake via TCP options.
📘 7. Nagle's Algorithm and Delayed ACKs
- Nagle's algorithm: Reduces the number of small packets by buffering data until either an ACK arrives or the segment reaches the MSS. Useful for interactive applications but can add latency.
- Delayed ACKs: The receiver delays sending an ACK for up to 500 ms to allow piggybacking on outgoing data. Reduces ACK overhead but can increase RTT and interfere with fast retransmit.
Both are trade‑offs between efficiency and latency.
📘 8. Modern TCP Implementations: Cubic, BBR
- TCP Cubic: Default in Linux. Uses a cubic function for window growth, which is more aggressive for large windows (improving high BDP performance) but also more fair because growth depends on time since last loss.
- TCP BBR (Bottleneck Bandwidth and RTT): Uses estimates of delivery rate and RTT to set the sending rate. Aims to achieve high throughput while maintaining low latency, by pacing packets to match the bottleneck bandwidth.
BBR does not rely on loss as a primary congestion signal; instead it measures the bandwidth and RTT directly.
📘 9. Performance Trade‑offs and Tuning
- Buffer sizes: Larger buffers can reduce loss but increase latency (bufferbloat).
- Initial cwnd: Larger initial windows (e.g., 10 MSS) improve short‑flow performance.
- TCP stack tuning: Parameters like
net.ipv4.tcp_rmem, net.ipv4.tcp_wmem, net.core.rmem_max affect performance.
- Disabling Nagle: For low‑latency applications, set
TCP_NODELAY.
📘 10. Case Studies: High‑Speed Networks, Data Centers
- High‑speed long‑distance networks: Require large windows and SACK to achieve high throughput. Cubic is well‑suited.
- Data centers: Low latency is critical. TCP variants like DCTCP use ECN to react quickly.
- Mobile networks: Variable RTT and loss require robust algorithms; BBR can adapt well.
📝 Quiz
Test your understanding with these 35 questions. Answers are hidden below each.
- What is the relationship between throughput, window size, and RTT?
Answer
Throughput = window / RTT (assuming window is the limiting factor).
- What is the bandwidth‑delay product (BDP)?
Answer
BDP = bandwidth × RTT; the amount of data that can be in transit.
- Why is it important for the TCP window to be at least the BDP?
Answer
To fully utilize the link; otherwise the link is underutilized.
- What is the maximum advertised window without window scaling?
Answer
65535 bytes (16‑bit window field).
- What is the purpose of TCP window scaling?
Answer
To allow windows larger than 65535 bytes by scaling the advertised window, essential for high BDP networks.
- What is SACK and how does it improve performance?
Answer
Selective Acknowledgment allows the receiver to acknowledge non‑contiguous blocks, enabling selective retransmission and reducing unnecessary retransmissions.
- What are TCP Timestamps used for?
Answer
For accurate RTT measurement and PAWS (Protection Against Wrapped Sequence numbers).
- What is Nagle's algorithm?
Answer
An algorithm that reduces small packet transmissions by buffering data until an ACK arrives or the segment reaches the MSS.
- When would you disable Nagle's algorithm?
Answer
For low‑latency applications like gaming or VoIP, where small packets need to be sent immediately (use TCP_NODELAY).
- What is a delayed ACK?
Answer
A receiver delays sending an ACK for up to 500 ms to allow piggybacking on outgoing data, reducing ACK overhead.
- How does delayed ACK affect RTT estimation?
Answer
It can increase the measured RTT, leading to larger RTOs and potentially slower recovery.
- What is TCP Cubic?
Answer
A congestion control algorithm that uses a cubic function for window growth, improving performance in high‑speed networks.
- What is TCP BBR?
Answer
A congestion control algorithm that uses bottleneck bandwidth and RTT estimates to set the sending rate, aiming for high throughput and low latency.
- How does BBR differ from Reno?
Answer
BBR measures bandwidth and RTT directly and paces packets; Reno reacts to loss and uses AIMD.
- What is bufferbloat?
Answer
Excessive buffering in routers causing high latency without improving throughput.
- How can bufferbloat be mitigated?
Answer
Using Active Queue Management (AQM) like CoDel or PIE, which manage queue lengths.
- What is the effect of a large initial cwnd?
Answer
It reduces latency for short flows by avoiding slow start.
- What is the effect of a small receiver buffer?
Answer
It limits the advertised window, reducing throughput.
- What is the relationship between window size and loss rate?
Answer
Larger windows lead to higher throughput but also increase the impact of losses (more data to retransmit).
- How does RTT affect fairness among TCP flows?
Answer
Flows with shorter RTTs get more bandwidth because they increase cwnd faster.
- What is ACK clocking?
Answer
The self‑pacing of TCP transmissions driven by the arrival of ACKs.
- Why is ACK clocking important?
Answer
It helps smooth traffic and avoid bursts.
- What is the send buffer (SO_SNDBUF) used for?
Answer
To hold data sent but not yet acknowledged; its size limits the amount of outstanding data.
- What is the receive buffer (SO_RCVBUF) used for?
Answer
To hold data received but not yet read by the application; its size limits the advertised window.
- How can you check the current TCP settings in Linux?
Answer
Using sysctl net.ipv4.tcp_* or ip route show.
- What is the PAWS mechanism?
Answer
Protection Against Wrapped Sequence numbers, using timestamps to reject old segments after sequence number wrap‑around.
- Why is PAWS necessary?
Answer
To prevent old duplicate segments from being accepted after the sequence number wraps.
- What is the typical value of the timestamp granularity in Linux?
Answer
Typically 1 millisecond or 1 microsecond, depending on the kernel.
- How does SACK interact with fast retransmit?
Answer
SACK provides more information, allowing the sender to retransmit only the missing data during fast recovery.
- What is the role of the
tcp_rmem and tcp_wmem parameters?
Answer
They set the minimum, default, and maximum receive and send buffer sizes.
- What is the effect of setting a very large receiver buffer?
Answer
It allows a large advertised window, increasing throughput, but may lead to memory consumption and bufferbloat.
- What is the difference between
SO_RCVBUF and the TCP advertised window?
Answer
SO_RCVBUF is the maximum buffer size; the advertised window is the current free space in that buffer.
- How does the TCP stack determine the MSS?
Answer
It uses the path MTU discovery (PMTUD) or default (typically 1460 for Ethernet).
- What is the benefit of using jumbo frames for TCP?
Answer
Larger MSS reduces overhead and increases efficiency, improving throughput.
- What is the impact of packet reordering on TCP performance?
Answer
Reordering can cause spurious fast retransmits, reducing performance; Timestamps and SACK can help.
🛠️ Exercises
Apply your knowledge with these 20 exercises. Solutions are provided below each.
- Exercise 1: Throughput Calculation
A TCP connection has window size = 64 KB, RTT = 50 ms. What is the maximum throughput?
Solution
Throughput = (64 * 1024 * 8) / 0.05 = 524,288 / 0.05 = 10,485,760 bps ≈ 10.49 Mbps.
- Exercise 2: BDP Calculation
A link has bandwidth = 100 Mbps, RTT = 30 ms. What is the BDP in bytes?
Solution
BDP = 100e6 * 0.03 = 3,000,000 bits = 375,000 bytes.
- Exercise 3: Window Scaling
If the advertised window is 65535 and the scaling factor is 4, what is the effective window size?
Solution
Effective = 65535 * 2^4 = 65535 * 16 = 1,048,560 bytes ≈ 1 MB.
- Exercise 4: Utilization with Window
For a link with BDP = 500 KB and window = 64 KB, what is the utilization?
Solution
Utilization = window / BDP = 64 KB / 500 KB = 0.128 = 12.8%.
- Exercise 5: SACK Benefit
Explain how SACK reduces the number of retransmissions when two packets are lost in a window of 10 packets.
Solution
Without SACK, the sender may retransmit all packets after the first loss (if cumulative ACKs). With SACK, it knows which packets were received and only retransmits the lost ones.
- Exercise 6: Nagle's Algorithm Effect
An application sends 10 bytes every 10 ms. Nagle is enabled. Describe the transmission pattern.
Solution
The first byte may be sent; subsequent bytes may be buffered until an ACK arrives or enough data accumulates, reducing the number of packets.
- Exercise 7: Delayed ACK Impact on RTO
If the delayed ACK timer is 200 ms and the actual RTT is 50 ms, how does this affect the RTO?
Solution
The measured RTT will be larger (e.g., 250 ms), increasing the RTO, which may reduce performance.
- Exercise 8: Cubic Window Growth
In TCP Cubic, how does the window grow after a loss?
Solution
It grows rapidly initially (like slow start) and then follows a cubic function, which is more aggressive for large windows.
- Exercise 9: BBR vs. Reno
Why does BBR not rely on loss to adjust its rate?
Solution
BBR uses bandwidth and RTT measurements directly, aiming to match the bottleneck rate without inducing loss.
- Exercise 10: Bufferbloat Mitigation
What is the role of CoDel in reducing bufferbloat?
Solution
CoDel monitors queue delay and drops packets when delay exceeds a threshold, keeping queues small.
- Exercise 11: Initial Window Size
With initial cwnd = 10 MSS, how many RTTs to reach 100 MSS (assuming no losses)?
Solution
Doubling: 10→20→40→80→160. So after 4 RTTs it reaches 160, so it reaches 100 sometime during the 4th RTT.
- Exercise 12: ACK Clocking
In a steady state, an ACK arrives every 10 ms. The MSS is 1460 bytes. What is the sending rate?
Solution
Rate = MSS / ACK_interval = 1460 * 8 / 0.01 = 1,168,000 bps ≈ 1.17 Mbps.
- Exercise 13: Timestamp Granularity
If the timestamp granularity is 1 ms, what is the maximum RTT that can be measured accurately?
Solution
It can measure up to 2^32 ms ≈ 49.7 days; but granularity limits precision to 1 ms.
- Exercise 14: SACK and Fast Recovery
How does SACK improve fast recovery compared to Reno?
Solution
SACK provides information about which packets have been received, allowing the sender to retransmit only the missing ones during recovery, rather than guessing.
- Exercise 15: Window Update
If the receiver's application reads data, how does that affect the advertised window?
Solution
It increases the free space, so the advertised window (rwnd) increases, allowing the sender to send more.
- Exercise 16: PAWS
Why is PAWS needed when sequence numbers can wrap around?
Solution
To prevent a duplicate segment with a valid sequence number (after wrap) from being accepted, using timestamps to ensure it is not too old.
- Exercise 17: TCP Tuning
For a high‑speed network, which parameters would you increase?
Solution
Increase the window size (via window scaling), increase send/receive buffers, and perhaps enable SACK and timestamps.
- Exercise 18: Impact of Loss on Throughput
Using the square‑root law, if loss rate doubles, what is the approximate throughput change?
Solution
Throughput becomes 1/√2 ≈ 0.707 times original, a 29.3% decrease.
- Exercise 19: RTT Unfairness
Two flows share a bottleneck. Flow A has RTT=20ms, Flow B has RTT=100ms. Which gets more bandwidth, and roughly by what factor?
Solution
Flow A gets more bandwidth, roughly by a factor of (100/20) ≈ 5 times, but other factors limit.
- Exercise 20: Data Center TCP
Why is DCTCP designed with ECN?
Solution
To react to congestion quickly and maintain low latency, which is critical in data centers.
📚 Homework
These advanced problems require synthesis, research, and quantitative analysis. Sample answers are provided below.
- Problem 1: Derive the optimal window size for a given link.
Given bandwidth B and RTT, show that the window size should be at least BDP to fully utilize the link. Prove that if window > BDP, it may cause queuing delay.
Sample Answer
Let W be the window in bytes. Throughput = W/RTT. To achieve B, W must be ≥ B*RTT = BDP. If W > BDP, packets will queue, increasing delay.
- Problem 2: Impact of Window Scaling on Performance.
Compare the throughput of a 1 Gbps, 100 ms RTT link with and without window scaling (assuming no losses).
Sample Answer
Without scaling, Wmax=64KB, throughput=64KB/0.1=5.24 Mbps. With scaling, W can be set to BDP=12.5MB, throughput=1 Gbps.
- Problem 3: SACK and Multi‑packet Loss.
Explain how SACK helps in a scenario where 3 packets are lost in a window of 10. Compare with cumulative ACK.
Sample Answer
Cumulative ACK would only ACK up to the first loss, causing retransmission of all subsequent packets. SACK can indicate the gaps, retransmitting only the three lost packets.
- Problem 4: Timestamp and RTT Measurement.
Describe how the Timestamp option improves RTT measurement compared to the basic method.
Sample Answer
Basic method uses the time of sending and ACK, but ACK may be delayed. Timestamp option includes the sender's timestamp in the ACK, allowing accurate RTT even with delayed ACKs.
- Problem 5: Nagle's Algorithm and Applications.
Compare the performance of a telnet session (interactive) and a file transfer with Nagle's algorithm. Which benefits and which suffers?
Sample Answer
File transfer benefits (fewer packets), telnet suffers (latency). Thus, telnet should disable Nagle.
- Problem 6: BBR vs. Cubic in a Lossy Environment.
Discuss the advantages of BBR over Cubic in a network with high packet loss.
Sample Answer
BBR does not rely on loss, so it can maintain high throughput even with loss, whereas Cubic reduces its window on loss.
- Problem 7: Bufferbloat Mitigation.
Research CoDel and describe how it works. How does it affect TCP performance?
Sample Answer
CoDel drops or marks packets based on queue delay; it keeps queues small, reducing latency while maintaining utilization.
- Problem 8: Window Update and Zero‑Window.
Explain the interactions between flow control (rwnd) and congestion control (cwnd) when rwnd is small.
Sample Answer
If rwnd < cwnd, the effective window is limited by rwnd. The sender may have to wait for window updates, reducing throughput.
- Problem 9: TCP Performance in Data Centers.
Why are data center TCP variants (e.g., DCTCP) different from standard TCP?
Sample Answer
They use ECN for early congestion notification, have lower RTTs, and require low latency and high burst tolerance.
- Problem 10: Impact of Packet Reordering.
Derive how packet reordering can cause spurious fast retransmits and reduce throughput. How do SACK and timestamps help?
Sample Answer
Reordering generates duplicate ACKs, which may trigger fast retransmit unnecessarily. SACK provides more information to avoid false positives; timestamps help distinguish order.
- Problem 11: TCP Tuning for 10 Gbps.
What kernel parameters would you adjust for a 10 Gbps link with RTT=10 ms?
Sample Answer
Set large buffer sizes (rmem, wmem), enable window scaling, SACK, timestamps, increase initial cwnd.
- Problem 12: ACK Clocking and Burstiness.
Explain how ACK clocking can lead to bursty traffic. How can pacing help?
Sample Answer
ACK clocking can cause bursts when ACKs arrive in groups. Pacing spreads out transmissions to smooth traffic.
- Problem 13: PAWS and Sequence Number Wrap.
At what bandwidth does sequence number wrap become a problem (assuming 32‑bit sequence numbers)?
Sample Answer
Wrap time = 2^32 * 8 / bandwidth (bits). For 1 Gbps, wrap time ≈ 34 seconds. Timestamps extend this.
- Problem 14: SACK vs. Cumulative ACK Overhead.
Compare the overhead of SACK options in the TCP header versus cumulative ACKs.
Sample Answer
SACK adds more data to the header (up to 40 bytes for 4 blocks), increasing overhead, but reduces retransmissions.
- Problem 15: TCP BBR and Fairness.
Discuss the fairness of BBR compared to Cubic. Are there any concerns?
Sample Answer
BBR aims to be fair, but in some scenarios it can be aggressive and take more bandwidth than Cubic. Ongoing research.
📌 Summary
- TCP throughput is limited by window size and RTT; the bandwidth‑delay product defines the required window for full utilization.
- Advanced features: SACK, Window Scaling, and Timestamps are essential for high‑performance networks.
- Nagle's algorithm and delayed ACKs are performance trade‑offs.
- Modern algorithms like Cubic and BBR improve performance in diverse environments.
- Performance tuning involves adjusting buffer sizes, window scaling, and other parameters.
In the next tutorial, we will explore QUIC, HTTP/3, and Modern Transport Protocols.