After completing this tutorial, you should be able to:
In Tutorial 4.11, we explored the overall architecture of IPsec, including Security Associations, the SPD and SAD, and the IKE protocol. We also introduced the two main protocols: Authentication Header (AH) and Encapsulating Security Payload (ESP). In this tutorial, we dive deep into these protocols, examining their packet formats, processing steps, and the cryptographic algorithms they use. Understanding the details of AH and ESP is essential for configuring IPsec, troubleshooting issues, and appreciating the trade-offs between the two protocols.
We begin with a recap of the services provided by AH and ESP. We then dissect the AH packet format, field by field, and explain how AH provides integrity and authentication by protecting the IP header and payload. We also describe the processing of AH on outbound and inbound packets. Next, we cover ESP in detail, including its header, trailer, and optional authentication fields. ESP provides confidentiality through encryption, and can also provide integrity and authentication when used with an integrity algorithm.
We discuss the cryptographic algorithms commonly used with IPsec, such as AES for encryption and SHA-2 for integrity, and the concept of combined mode (e.g., AES-GCM) that provides both encryption and authentication in a single algorithm. We illustrate the packet structures for both transport and tunnel modes, showing how the original packet is transformed. We then explain the inbound and outbound processing steps in detail, including the anti-replay mechanism using sequence numbers and sliding windows.
We also address performance considerations, including the overhead of encryption and the impact of different algorithms on throughput. Finally, real-world case studies illustrate practical deployments and common pitfalls. By the end of this tutorial, you will have a detailed understanding of how AH and ESP operate, enabling you to configure and troubleshoot IPsec implementations effectively.
This content aligns with Stallings & Brown (2024), Chapter 16 and IETF RFCs 4302 (AH) and 4303 (ESP).
Historically, AH was used when encryption was not required or allowed (e.g., due to export restrictions). Today, ESP with authentication is almost always preferred because it provides a complete security solution. AH is rarely used in modern networks, except in some legacy or specialized scenarios where header authentication is specifically needed and encryption is prohibited. ESP in tunnel mode is the standard for VPNs.
The AH header is inserted after the original IP header (in transport mode) or after the new IP header (in tunnel mode). Its format is defined in RFC 4302. The fields are:
Figure 1: AH Packet Format
AH protects the IP header fields that do not change in transit (e.g., source/destination IP, protocol, version, and some others) and the entire payload. Mutable fields like TTL, checksum, and options are set to zero before the MAC computation. This provides authentication of the source and ensures the packet has not been tampered with.
ESP is defined in RFC 4303. It consists of a header, payload, trailer, and optional authentication data. The format varies depending on whether authentication is used and whether the mode is transport or tunnel. The main fields are:
Figure 2: ESP Packet Format (with authentication)
ESP can provide integrity and authentication by including the Authentication Data field. In modern deployments, it is common to use AEAD (Authenticated Encryption with Associated Data) algorithms like AES-GCM or ChaCha20-Poly1305, which provide both encryption and authentication in a single pass, reducing complexity and improving performance. This is often called "combined mode" and is the recommended practice.
IKE negotiates the algorithms to be used for the IPsec SA. It is important to configure a policy that only allows strong algorithms and disallows weak ones (e.g., NULL encryption, MD5, SHA-1, DES). The IETF guidelines (e.g., BCP 195) provide recommendations.
In practice, ESP is almost always used with authentication. There are two approaches:
The AEAD approach is recommended by NIST and IETF, and is the default in most modern IPsec implementations. It simplifies configuration and reduces the risk of misconfiguration.
In transport mode, the original IP header is retained. The IPsec header (AH or ESP) is inserted after it. The payload (TCP/UDP segment) is protected.
In tunnel mode, a new IP header is added, and the entire original IP packet is treated as the payload. The new header may have different source/destination IPs (e.g., gateway addresses).
Tunnel mode has more overhead due to the additional IP header. Transport mode is more efficient but less flexible. For VPNs, tunnel mode is standard.
The processing steps for IPsec are detailed in RFC 4301. Below is a simplified overview.
If any step fails (e.g., ICV mismatch, sequence number out of window), the packet is discarded and an error may be logged. This is critical for security.
Each IPsec packet includes a 32-bit sequence number in the AH or ESP header. For each SA, the sender increments a counter and includes it in each packet. The receiver maintains a sliding window of acceptable sequence numbers.
The receiver's window is typically 32, 64, or 128 packets. When a packet arrives, the receiver checks if the sequence number is greater than the highest received number minus the window size. If the number is within the window and not already received, the packet is accepted; otherwise, it is discarded. This prevents replay attacks where an attacker resends a captured packet.
When the sequence number reaches 2^32 - 1, the SA must be renegotiated (or a new SA established) to avoid reuse. This is handled by IKE.
Cryptographic operations and IKE negotiations introduce latency. For real-time applications (e.g., VoIP), this can be a concern. Using faster algorithms and hardware acceleration helps.
An organization used AH in transport mode for securing internal server communications. They encountered issues with NAT and performance. They migrated to ESP with AES-GCM in transport mode, which resolved NAT issues (ESP is more NAT-friendly) and improved performance due to the combined mode. The migration also simplified configuration.
A global corporation deployed ESP tunnel mode with AES-256-GCM and SHA-256 for site-to-site VPNs between data centers. They enabled PFS and used IKEv2 with certificate-based authentication. The VPNs provided high throughput (10 Gbps) with hardware-accelerated encryption, ensuring secure and high-performance connectivity.
A company experienced packet loss on their IPsec VPN. Investigation revealed that the anti-replay window was set too small (32 packets) for their high-traffic environment, causing valid packets to be dropped. Increasing the window to 128 packets resolved the issue. This highlights the need to tune anti-replay settings based on traffic patterns.
This tutorial provided a detailed exploration of the two main IPsec protocols: Authentication Header (AH) and Encapsulating Security Payload (ESP). We examined the packet formats of both protocols, including the fields and their purposes. AH provides integrity and authentication by protecting the IP header and payload, but does not encrypt. ESP provides confidentiality through encryption, and can also provide integrity and authentication when used with an integrity algorithm or in combined mode (AEAD).
We covered the processing steps for both outbound and inbound packets, highlighting the role of the SPD and SAD, the sequence number for anti-replay, and the verification of the ICV. We discussed the anti-replay mechanism in detail, including the sliding window and sequence number management.
We compared transport and tunnel modes, illustrating the packet structures and use cases for each. We also reviewed cryptographic algorithms recommended for IPsec, emphasizing the importance of using strong algorithms (AES-GCM, SHA-256) and avoiding weak ones. The combined mode with AEAD was presented as the modern best practice.
We addressed performance considerations, including the overhead of encryption and the impact on throughput and latency, and provided optimization strategies. Real-world case studies demonstrated the practical application of AH and ESP, as well as common pitfalls and lessons learned.
By the end of this tutorial, you should be able to configure and troubleshoot IPsec using AH and ESP, select appropriate algorithms, and understand the trade-offs involved. The next tutorial will build on this by exploring virtual private networks (VPNs) and secure remote access in depth.
Next: Tutorial 4.13: Virtual Private Networks and Secure Remote Access.
1. Which IPsec protocol provides confidentiality (encryption)?
2. In the AH header, what field is used to identify the Security Association?
3. Which of the following is NOT provided by AH?
4. What is the purpose of the Sequence Number field in AH and ESP?
5. In ESP, the Authentication Data field is used for:
6. Which mode of IPsec is typically used for site-to-site VPNs?
7. Which algorithm is an example of AEAD (Authenticated Encryption with Associated Data) commonly used in ESP?
8. What happens when the sequence number in an IPsec SA reaches 2^32 - 1?
9. In ESP, the Pad Length field indicates:
10. Which field in the AH header is set to zero before computing the ICV?
11. What is the main advantage of using ESP combined mode (AEAD) over separate encryption and integrity algorithms?
12. Which of the following is a limitation of AH compared to ESP?
Exercise 1: Packet Format Analysis Intermediate
Draw the packet structure for an ESP packet in tunnel mode with authentication (using AES-GCM). Label each field with its size and purpose. Then, draw the structure for AH in transport mode and explain the differences.
ESP Tunnel with AES-GCM:
AH Transport:
Key differences: ESP encrypts, AH doesn't; ESP adds padding, AH doesn't; AH protects the IP header, ESP doesn't (except in tunnel).
Exercise 2: Anti-Replay Window Calculation Intermediate
An IPsec receiver has an anti-replay window of size 64. The last valid sequence number received was 1000. A packet arrives with sequence number 950. Is it accepted? What if it is 1020? Explain the logic.
The window covers numbers from (1000 - 64 + 1) = 937 to 1000. So, a packet with seq 950 is within the window and may be accepted if not already seen. Seq 1020 is greater than 1000, which is beyond the window; it would be accepted as long as it's not a replay, and the window would shift to cover 957 to 1020. However, if seq 950 is less than 937, it would be rejected.
Exercise 3: Algorithm Selection Introductory
You are configuring a new IPsec VPN. Which combination of algorithms would you choose: (AES-CBC + HMAC-SHA-256) or (AES-GCM)? Explain your choice.
Choose AES-GCM. It is an AEAD algorithm that provides both encryption and authentication in one pass, reducing overhead and complexity. It is also more secure against certain attacks (e.g., padding oracle). AES-GCM is recommended by NIST and IETF.
Exercise 4: Processing Order Advanced
For an outbound ESP packet with separate encryption (AES-CBC) and authentication (HMAC-SHA-256), describe the order of operations. Does the packet get encrypted first, then authenticated, or vice versa? Explain why.
The order is: encrypt first, then authenticate (compute ICV over the encrypted payload). This is because the authentication covers the encrypted data to detect tampering with the ciphertext. If authentication were done first, the encryption would change the data and the authentication would fail. In practice, AEAD is preferred to avoid this complexity.
Exercise 5: Configuration Scenario Intermediate
A network administrator wants to set up IPsec between two hosts for secure management (SSH). They decide to use AH in transport mode. What are the implications of this choice? Would ESP be better? Explain.
Using AH in transport mode provides authentication and integrity but no encryption. SSH itself provides encryption, so AH adds authentication of the IP headers. However, AH does not work with NAT, which may be a problem if the management path involves NAT. ESP with null encryption could be used, but modern recommendations are to use ESP with encryption (e.g., AES-GCM) to protect both the IP payload and the transport header. Since SSH already encrypts the payload, ESP in transport mode with encryption may be redundant but still adds integrity and authentication for the IP header if combined mode is used.
Homework 1: Write a 2,000-word research paper comparing the security and performance characteristics of ESP with AES-CBC + HMAC-SHA-256 vs. ESP with AES-GCM. Discuss the cryptographic differences, potential vulnerabilities, and performance benchmarks. Include references.
Key points: AES-CBC requires padding, is susceptible to padding oracle attacks if not used with integrity (but with HMAC, it's mitigated); AES-GCM is a stream cipher with integrated authentication, faster in hardware (AES-NI), and simpler. Performance: AES-GCM is generally faster because it combines operations. Security: both are secure when used properly, but AES-GCM is recommended for new deployments.
Homework 2: Using a packet capture tool (Wireshark) or a network simulator, capture IPsec traffic (ESP) and analyze the packet structure. Identify the SPI, sequence number, encrypted payload, and ICV. Write a lab report with screenshots and explanations.
Practical assignment; report should include setup of IPsec (e.g., using strongSwan or a VPN), capture of traffic, and analysis of ESP packets showing the headers and encrypted data.
Homework 3: Investigate the NAT traversal mechanisms for IPsec (NAT-T). Explain how ESP can work through NAT using UDP encapsulation. Write a 1,500-word paper covering the challenges, the solution (RFC 3947/3948), and the configuration steps.
Key points: NAT changes IP addresses, which breaks AH and ESP because the IP header or SPI may be modified. NAT-T encapsulates ESP in UDP (port 4500) and uses IKE to negotiate NAT presence. The UDP header is not protected, allowing NAT to change addresses. The encapsulation adds overhead but enables IPsec through NAT.
Homework 4: Design an IPsec policy for a secure connection between a branch office and the headquarters. Specify the mode (transport/tunnel), the protocols (AH/ESP), the algorithms (encryption, integrity), key management (IKE), and anti-replay parameters. Justify your choices.
Policy: Use ESP in tunnel mode with AES-256-GCM (combined mode), IKEv2 with certificate-based authentication, PFS enabled, sequence number anti-replay with window size 128. Justification: tunnel mode protects all traffic; AES-GCM is efficient and secure; IKEv2 is robust; PFS ensures forward secrecy.
Homework 5: Research the vulnerability "CRIME" or "BEAST" (or other attacks on TLS) and discuss whether a similar attack could apply to IPsec. Write a case study analyzing the risks and mitigation.
Key points: BEAST and CRIME are attacks on TLS/SSL that exploit CBC mode and compression. IPsec with ESP using CBC mode may be vulnerable to similar attacks if not properly implemented, but the risk is mitigated by using AEAD algorithms (AES-GCM) and disabling compression.
COMP400 – Computer and Network Security (Revision 3) • Unit 4: Security Systems and Models