Tutorial 3.6: Authentication Protocols and Mechanisms

← Previous Tutorial (3.5) | Table of Contents | Unit Home | Next Tutorial (3.7) →

Table of Contents

Learning Objectives

After completing this tutorial, you should be able to:

Overview

In the previous tutorials, we examined authentication factors—passwords, tokens, biometrics—and multi-factor authentication. However, authentication is rarely performed in isolation. In distributed systems, authentication occurs over networks where messages can be intercepted, altered, or replayed. This tutorial addresses the protocols and mechanisms that secure the authentication process itself, protecting it against a range of active and passive attacks.

We begin by defining what an authentication protocol is and why simple password transmission over a network is insecure. We then introduce the challenge-response paradigm, a foundational technique where the verifier sends a random challenge that the claimant must transform using a secret (e.g., a password or cryptographic key) and return the response. This prevents eavesdroppers from replaying captured credentials.

We explore mutual authentication, where both parties prove their identities to each other—a requirement for client-server interactions where both endpoints need assurance of the other's legitimacy. We examine the cryptographic primitives and message flows that enable mutual authentication.

A major focus is replay protection. We analyze the use of nonces (numbers used once), timestamps, and sequence numbers to ensure that captured authentication messages cannot be reused by an attacker. We discuss the trade-offs and security implications of each approach.

We then turn to session keys and tickets. Authentication protocols often establish a temporary session key for subsequent communication, reducing the risk of key compromise. Tickets (as used in Kerberos) allow a trusted third party to vouch for a user's identity, enabling single sign-on and delegation.

The tutorial provides a comprehensive security analysis of authentication protocols. We examine classic attacks: replay attacks, man-in-the-middle attacks, reflection attacks, and parallel session attacks. We discuss how protocol design choices—such as the order of messages, the use of encryption, and freshness guarantees—affect security.

We survey common authentication protocols, including Password Authentication Protocol (PAP), Challenge-Handshake Authentication Protocol (CHAP), and the Extensible Authentication Protocol (EAP). We compare their security properties and typical use cases.

The tutorial concludes with two case studies: one on the famous Needham-Schroeder protocol and its vulnerability, and another on the WPA2 4-way handshake, illustrating how authentication protocols are used in wireless security. These case studies demonstrate the importance of rigorous protocol analysis and the consequences of design flaws.

By the end of this tutorial, you will have a deep understanding of authentication protocols, enabling you to evaluate, design, and troubleshoot secure authentication mechanisms in real-world systems. This foundation is essential for Tutorial 3.7, where we will examine the Kerberos authentication architecture in detail.

1. Introduction to Authentication Protocols

1.1 What Is an Authentication Protocol?

An authentication protocol is a sequence of messages exchanged between two or more parties to verify the identity of one or more of the parties. The protocol defines:

Unlike simple password submission over a network, authentication protocols are designed to resist eavesdropping, replay, and other attacks. They often involve the exchange of nonces, timestamps, and cryptographic proofs.

1.2 Why Protocols Are Necessary

Consider a naive authentication scheme: the client sends its password in cleartext to the server. This is insecure because:

Authentication protocols address these vulnerabilities by ensuring that the authentication information is never transmitted in a reusable form, and that the freshness of the exchange is verified.

1.3 Desirable Properties of Authentication Protocols

Key Takeaway: Authentication protocols define the secure exchange of credentials over a network. They must provide freshness, mutual authentication, and resistance to replay and MITM attacks.

2. Challenge-Response Authentication

2.1 The Basic Concept

Challenge-response authentication is a family of protocols where the verifier sends a random challenge to the claimant. The claimant computes a response using a secret (e.g., a password hash or cryptographic key) and sends it back. The verifier checks the response to authenticate the claimant.

The key security property is that the challenge is fresh and unpredictable, so the response cannot be replayed. An eavesdropper who captures a response cannot use it for a different challenge.

Client (C) Server (S) ────────────────────────────────────────────────────────────────── 1. C ──────────────────────────────► S (Request access) 2. C ◄───────────────────────────── S (Challenge R) 3. C ────► f(secret, R) ──────────► S (Response) 4. C ◄───────────────────────────── S (Accept / Reject)

Figure 1: Basic challenge-response authentication flow.

2.2 Types of Challenge-Response

2.3 CHAP (Challenge-Handshake Authentication Protocol)

CHAP is a widely used challenge-response protocol for Point-to-Point Protocol (PPP) connections. It uses a shared secret (password) and a nonce (challenge) to authenticate. The response is computed as:

Response = Hash(Challenge || Password)

Where Hash is typically MD5. The server stores the password (or a hash) and verifies the response. CHAP provides replay protection because the challenge changes with each authentication attempt.

Security considerations:

2.4 Advantages and Limitations of Challenge-Response

Advantages:

Limitations:

Practical Note: Challenge-response is the foundation of many authentication protocols. It is often combined with mutual authentication and session key establishment to create robust authentication systems.

3. Mutual Authentication

3.1 The Need for Mutual Authentication

In many scenarios, both the client and the server need to verify each other's identity. For example, a client accessing a banking website wants to ensure that the server is legitimate (not a phishing site), and the server wants to ensure the client is legitimate. Mutual authentication provides this two-way verification.

3.2 Two-Way Challenge-Response

Mutual authentication can be achieved by performing two challenge-response exchanges, one in each direction.

Client (C) Server (S) ────────────────────────────────────────────────────────────────── 1. C ──────────────────────────────► S (Request access) 2. C ◄───────────────────────────── S (Challenge C1) 3. C ────► Response to C1 ─────────► S (Authenticate C to S) 4. C ◄───────────────────────────── S (Challenge C2) 5. C ────► Response to C2 ─────────► S (Authenticate S to C) 6. C ◄───────────────────────────── S (Accept / Reject)

Figure 2: Mutual authentication with two challenge-response exchanges.

3.3 The Needham-Schroeder Protocol

The Needham-Schroeder protocol is a classic mutual authentication protocol that uses a trusted third party (TTP) to establish a session key. It is an early example of a protocol that provides mutual authentication and key distribution. We will examine its vulnerability in Section 6.

3.4 Public-Key Mutual Authentication

Using public-key cryptography, mutual authentication can be achieved by having each party sign a nonce and exchange the signed messages. This is the basis of TLS mutual authentication (client certificate authentication).

Key Takeaway: Mutual authentication ensures that both parties are legitimate. It is essential in environments where both endpoints need to trust each other, such as banking, e-commerce, and secure communications.

4. Replay Protection and Nonces

4.1 The Replay Attack

A replay attack occurs when an attacker captures a valid authentication message and re-transmits it later to impersonate the legitimate party. Even if the message is encrypted, if it is replayed exactly, the server may accept it. This is why replay protection is critical.

4.2 Nonces: Numbers Used Once

A nonce is a random or pseudo-random number that is used only once in a cryptographic protocol. Nonces are the most common mechanism for replay protection. The verifier generates a nonce and includes it in the challenge; the claimant includes it in the response. If an attacker replays an old response, the nonce will not match the current challenge.

Properties of a good nonce:

4.3 Timestamps as Freshness Guarantees

Instead of nonces, some protocols use timestamps to ensure freshness. The claimant includes a timestamp in the message, and the verifier checks that the timestamp is within an acceptable window. This requires synchronized clocks and can be vulnerable to clock skew attacks.

Comparison:

4.4 Sequence Numbers

Some protocols use sequence numbers that increment with each message. The receiver checks that the sequence number is increasing and within an expected range. Sequence numbers are stateful and require the parties to maintain counters.

4.5 Comparison Table

Mechanism Freshness Guarantee State Required Clock Sync Required Resistance to Replay
Nonce Unique per session Verifier must track used nonces No High
Timestamp Time window None (window check) Yes Moderate (clock skew)
Sequence Number Monotonic order Both parties maintain counters No High

5. Session Keys and Tickets

5.1 The Need for Session Keys

Authentication protocols often establish a session key—a temporary symmetric key used to encrypt subsequent communication. Using a session key provides:

5.2 Key Distribution

Session keys can be distributed in several ways:

5.3 Tickets

A ticket is a message that contains an identity and a session key, encrypted with a key known only to a trusted party. Tickets are used in protocols like Kerberos to enable single sign-on. A ticket allows a client to authenticate to a service without re-entering credentials.

Important: Session keys and tickets are fundamental to scalable authentication systems. They reduce the number of times a user must enter their credentials and enable secure, efficient communication.

6. Protocol Security Analysis

6.1 Common Attacks on Authentication Protocols

6.2 The Needham-Schroeder Vulnerability

The Needham-Schroeder protocol (public-key version) was considered secure for years until a flaw was discovered by Lowe (1995). The flaw allowed a MITM attack:

The fix was to include the intended recipient's identity in the message, e.g., {N, B}K_A instead of {N}K_A. This is a classic lesson in protocol design: seemingly small omissions can have serious security consequences.

6.3 Formal Analysis Methods

Authentication protocols are often verified using formal methods such as:

Key Takeaway: Authentication protocols must be carefully designed and analyzed to resist a range of attacks. Formal verification tools are essential for detecting subtle flaws that might be missed by manual analysis.

7. Common Authentication Protocols

7.1 PAP (Password Authentication Protocol)

7.2 CHAP (Challenge-Handshake Authentication Protocol)

7.3 EAP (Extensible Authentication Protocol)

7.4 Comparison Table

Protocol Authentication Type Mutual Authentication Replay Protection Key Establishment Security Level
PAP Password (cleartext) No No No Very low
CHAP Challenge-response (hashed password) No Yes (nonce) No Low to moderate
EAP-TLS Certificate-based (PKI) Yes Yes (nonces) Yes High
PEAP Tunneled authentication (password/MSCHAPv2) Yes (server cert) Yes Yes High (with tunnel)
Kerberos Ticket-based (TTP) Yes Yes (timestamps) Yes High

8. Case Studies

8.1 Case Study: Needham-Schroeder Protocol Flaw

Protocol: The Needham-Schroeder public-key protocol was published in 1978 and was widely used as a reference for mutual authentication. It was believed to be secure for many years.

Flaw: Lowe (1995) discovered a man-in-the-middle attack. The protocol did not include the recipient's identity in the encrypted nonce. As a result, an attacker could intercept a message from A to B and, in a separate session, impersonate B to A.

Consequence: This flaw led to a re-evaluation of protocol design principles and highlighted the importance of formal verification.

8.2 Case Study: WPA2 4-Way Handshake

Protocol: The 4-way handshake is used in WPA2 (Wi-Fi Protected Access II) to authenticate a client (supplicant) to an access point (authenticator) and establish a session key (PTK). It uses an 802.1X authentication server (RADIUS) or a pre-shared key (PSK).

Security: The handshake uses a four-message exchange with nonces (SNonce, ANonce) and MIC (Message Integrity Code) to verify the pairwise master key (PMK). It provides mutual authentication and key establishment.

Vulnerability (KRACK): In 2017, the KRACK attack (Key Reinstallation Attack) exploited a flaw in the 4-way handshake. The handshake allowed a client to reinstall an already-in-use session key, enabling the attacker to decrypt and inject packets. This vulnerability was caused by an implementation flaw in the state machine of the handshake, not the cryptographic primitives themselves.

Lesson: Even robust protocols must be implemented correctly. The KRACK attack demonstrated the importance of rigorous protocol state management and the potential for implementation vulnerabilities to compromise security.

9. Summary and Transition

This tutorial provided a comprehensive exploration of authentication protocols and mechanisms. We began by defining authentication protocols and their essential properties: freshness, mutual authentication, key establishment, and resistance to attacks. We examined challenge-response authentication as a foundational technique, providing replay protection by using fresh, unpredictable challenges.

Mutual authentication was discussed as a requirement for many modern systems, and we explored two-way challenge-response and public-key-based mutual authentication. Replay protection mechanisms were analyzed in detail, including nonces, timestamps, and sequence numbers, with a comparison of their trade-offs.

Session keys and tickets were introduced as mechanisms for establishing secure communication channels and enabling single sign-on. We then conducted a security analysis of authentication protocols, examining common attacks such as replay, MITM, reflection, and parallel session attacks. The Needham-Schroeder vulnerability illustrated how subtle flaws can undermine a protocol's security.

We surveyed common authentication protocols—PAP, CHAP, and EAP—and compared their security properties. The case studies on Needham-Schroeder and the WPA2 4-way handshake demonstrated the real-world implications of protocol design and implementation flaws.

This tutorial has equipped you with the knowledge to evaluate, design, and troubleshoot authentication protocols. In the next tutorial, Tutorial 3.7, we will focus on the Kerberos authentication architecture, a widely used protocol for network authentication that leverages many of the concepts we have covered—challenge-response, mutual authentication, session keys, and tickets.

Quiz

Answer the following questions to check your understanding. Click the "Answer" button to reveal the solution.

Q1. Which of the following is not a desirable property of an authentication protocol?

Answer
D) A fixed nonce would provide no freshness and would be vulnerable to replay attacks. Nonces must be unique and unpredictable.

Q2. In a challenge-response authentication protocol, the verifier sends a challenge to the claimant. The primary purpose of the challenge is to:

Answer
B) The challenge ensures freshness and prevents replay attacks because the response is bound to the specific, unpredictable challenge.

Q3. Which of the following protocols transmits passwords in cleartext?

Answer
A) PAP (Password Authentication Protocol) transmits passwords in cleartext.

Q4. Which of the following is a replay protection mechanism that requires synchronized clocks?

Answer
B) Timestamps require synchronized clocks to verify that the message is fresh.

Q5. What is the main security problem with the original Needham-Schroeder public-key protocol?

Answer
B) The lack of explicit identity binding in the encrypted nonce allowed a man-in-the-middle attack.

Q6. The KRACK attack on WPA2 exploited a vulnerability in:

Answer
B) The KRACK attack exploited an implementation flaw in the 4-way handshake state machine, allowing reinstallation of a session key.

Q7. A ticket in an authentication system like Kerberos is primarily used to:

Answer
B) A ticket proves the user's identity to a service and enables single sign-on.

Q8. Which EAP method provides the strongest security (mutual authentication with certificates)?

Answer
B) EAP-TLS uses client and server certificates for mutual authentication, providing the highest security.

Q9. A reflection attack occurs when an attacker:

Answer
B) In a reflection attack, the attacker reflects a message back to the sender, tricking the sender into believing they are communicating with another party.

Q10. Which of the following is a key benefit of establishing a session key during authentication?

Answer
B) Session keys provide forward secrecy: even if long-term keys are compromised, past session keys remain secure.

Q11. CHAP uses a hash function (e.g., MD5) to compute the response. What is the primary security concern with using fast hash functions for password-based challenge-response?

Answer
B) Fast hash functions enable attackers to perform offline dictionary attacks on captured challenge-response pairs.

Q12. Mutual authentication is important because:

Answer
A) Mutual authentication ensures that both client and server verify each other's identities, preventing phishing and impersonation attacks.

Exercises

These exercises are designed to help you apply the concepts from this tutorial. Attempt each exercise before revealing the sample solution.

Exercise 3.6-1: Protocol Design

Design a simple challenge-response authentication protocol for a client-server system that uses a pre-shared secret key (K). The protocol should provide mutual authentication and establish a session key. Assume the server has a copy of K for each client. Provide a detailed message flow and explain how your protocol resists replay, MITM, and reflection attacks.

Sample Solution

Protocol flow:

Client (C) Server (S) ────────────────────────────────────────────────────────────────── 1. C ──────────────────────────────► S (Request, Nonce_C) 2. C ◄───────────────────────────── S (Nonce_S, {Nonce_C}K) 3. C ────► {Nonce_S}K, {SessionKey}K ──► S 4. C ◄───────────────────────────── S (Accept / Reject)
  • Step 1: Client sends a request with a nonce (Nonce_C) to ensure freshness.
  • Step 2: Server responds with its own nonce (Nonce_S) and an encrypted nonce_C using K. This authenticates the server to the client (since only S knows K).
  • Step 3: Client decrypts Nonce_C, verifies it matches, then sends Nonce_S encrypted with K (authenticating client to server) and the session key encrypted with K. This establishes a session key.
  • Step 4: Server verifies Nonce_S and responds with success or failure.
  • Replay protection: Nonces are fresh, so responses cannot be replayed.
  • MITM: The use of K ensures that only the legitimate parties can decrypt and respond correctly.
  • Reflection: The protocol uses separate nonces for each party and direction, preventing reflection attacks.

Exercise 3.6-2: Vulnerability Analysis

Consider the following authentication protocol:

C → S: C, {C, N}K_CS S → C: {N, K_session}K_CS

Where C is the client identity, N is a nonce, and K_CS is a shared secret key between C and S. The protocol does not include any other messages or verification. Analyze this protocol for the following:

  1. Does it provide mutual authentication?
  2. Is it vulnerable to replay attacks?
  3. Is it vulnerable to reflection attacks?
  4. Is the session key distribution secure?
  5. Propose improvements to make it secure.
Sample Solution
  • Mutual authentication: No. The protocol authenticates the client (C) to the server (S) because C encrypts the nonce with K_CS. However, S does not authenticate itself to C—there is no proof that S knows K_CS (S simply responds with the same nonce). An attacker could impersonate S.
  • Replay attacks: Vulnerable. An attacker could capture the first message (C, {C,N}K_CS) and replay it to impersonate C. The nonce N is not fresh in the first message; it is chosen by C, not S. A fresh challenge from S is needed.
  • Reflection attacks: Potentially vulnerable. An attacker could reflect the first message back to C to get a response.
  • Session key distribution: The session key is encrypted with K_CS, which is secure. However, there is no confirmation that the client received the key, and the session key is not bound to a fresh nonce from the server.
  • Improvements:
    • Use a server-chosen nonce (challenge) in the first message.
    • Include the server's identity in the second message to authenticate S.
    • Use mutual challenge-response with nonces from both sides.
    • Add a message where C confirms receipt of the session key.

Exercise 3.6-3: Comparing CHAP and EAP-TLS

Compare CHAP and EAP-TLS in terms of security, deployment complexity, and typical use cases. For each protocol, identify:

  1. The authentication mechanism (e.g., shared secret, certificates).
  2. Whether mutual authentication is provided.
  3. How replay attacks are prevented.
  4. How session keys are established.
  5. At least one scenario where each protocol would be appropriate.
Sample Solution
  • CHAP:
    • Mechanism: Challenge-response with a shared secret (password).
    • Mutual authentication: No (only client to server).
    • Replay protection: Uses a fresh challenge (nonce).
    • Session key: No session key established; used for authentication only.
    • Scenario: Point-to-point connections (PPP) in legacy VPNs or dial-up networks.
  • EAP-TLS:
    • Mechanism: Public-key certificates (PKI) for both client and server.
    • Mutual authentication: Yes, both client and server authenticate.
    • Replay protection: Uses nonces in the TLS handshake.
    • Session key: Establishes a master session key (MSK) used for encryption.
    • Scenario: Enterprise wireless networks (802.1X) where a PKI is deployed.
  • Comparison: EAP-TLS is more secure and provides key establishment, but requires a PKI and more infrastructure. CHAP is simpler but less secure.

Exercise 3.6-4: Protocol State Machine Analysis

Consider the following authentication protocol state machine for a server:

What are the potential vulnerabilities in this state machine? How could an attacker exploit the transition between states? Propose improvements to the state machine to prevent these vulnerabilities.

Sample Solution
  • Vulnerability 1: Replay attack. An attacker could capture a valid response and replay it in a new session. The state machine does not track the nonce used for each session, so the server would accept the replayed response.
  • Vulnerability 2: Session hijacking. If the attacker can guess the nonce (or if the nonce is predictable), they could craft a valid response.
  • Vulnerability 3: Parallel session attack. The server does not tie the response to a specific session, so an attacker could use information from one session to authenticate in another.
  • Improvements:
    • Store the nonce associated with each session ID. The response must include the session ID.
    • Use strong random nonce generation.
    • Implement a timeout for the challenge state; if no response is received, return to idle.
    • Ensure the protocol includes mutual authentication and session key binding.

Exercise 3.6-5: Kerberos Pre-Authentication

Kerberos can use pre-authentication to prevent password guessing attacks against the AS (Authentication Server). Describe how pre-authentication works, why it is needed, and what attacks it prevents. Compare pre-authentication with the basic Kerberos exchange (without pre-authentication).

Sample Solution
  • Need for pre-authentication: In basic Kerberos, a client sends an AS request with the client's identity. The AS responds with a TGT encrypted with the client's password-derived key. If an attacker intercepts this response, they can perform an offline dictionary attack by attempting to decrypt the TGT with guessed passwords. Pre-authentication prevents this by requiring the client to prove knowledge of the password before the AS sends the TGT.
  • How pre-authentication works: The client sends an AS request with a timestamp encrypted with the client's password-derived key. The AS decrypts the timestamp and checks that it is within an acceptable window. If the timestamp is valid, the AS sends the TGT. If not, the AS rejects the request.
  • Attacks prevented: Pre-authentication prevents offline dictionary attacks on the TGT. The attacker cannot perform a dictionary attack because they never obtain an encrypted TGT.
  • Comparison: Without pre-authentication, the AS response contains the TGT and is vulnerable to offline cracking. With pre-authentication, the AS checks a time-based proof of password possession before issuing the TGT, significantly improving security.

Homework

These homework questions require deeper analysis, research, and application. Answer each question comprehensively.

Homework 3.6-1: Formal Analysis of a Protocol

Choose one of the following protocols: Needham-Schroeder (public-key), Otway-Rees, or Woo-Lam. Write a 1,000–1,250 word analysis that includes:

  1. A detailed description of the protocol messages and the security goals.
  2. An analysis of the protocol's security properties (mutual authentication, key secrecy, resistance to attacks).
  3. A description of any known vulnerabilities and how they were discovered.
  4. If vulnerabilities exist, propose fixes and explain how they address the flaws.
  5. A discussion of the lessons learned for protocol design.
Sample Answer

Analysis of the Needham-Schroeder Public-Key Protocol

  • Protocol description: The protocol establishes mutual authentication and a session key between two parties (A and B) using a trusted server (S). It uses public-key cryptography and nonces.
  • Security goals: Mutual authentication, key establishment, and resistance to replay and MITM attacks.
  • Known vulnerabilities: Lowe (1995) discovered a MITM attack due to the absence of explicit identity binding in the encrypted nonces. An attacker can impersonate B to A and A to B.
  • Fixes: Include the intended recipient's identity in the encrypted nonce (e.g., {N, B}K_A). This prevents the MITM attack.
  • Lessons: Protocol design must include explicit identities in encrypted messages to prevent reflection and MITM attacks. Formal verification is essential.

Homework 3.6-2: Designing a Secure Authentication Protocol

Design a secure authentication protocol for a distributed system with three entities: a client (C), a server (S), and a trusted third party (T). The protocol should provide:

Provide a detailed message flow, the cryptographic operations, and an explanation of how your protocol meets each security requirement. Also, discuss any assumptions you make about the security of T.

Sample Answer

Protocol Design with Trusted Third Party

Assumptions: C and S each share a long-term secret key with T (K_CT, K_ST). T is trusted and secure. All parties have synchronized clocks (or can tolerate a small time window).

Message flow:

Client (C) Trusted Third Party (T) Server (S) ────────────────────────────────────────────────────────────────────────── 1. C ──► T: Request, ID_C, ID_S, Nonce_C 2. T ──► C: {Ticket, Nonce_C, ID_S}K_CT 3. C ──► S: Ticket, {Nonce_S, ID_C}K_session 4. S ──► C: {Nonce_C, ID_S}K_session
  • Step 1: C requests a ticket from T, providing its identity, S's identity, and a nonce.
  • Step 2: T generates a session key K_session and creates a ticket for S (encrypted with K_ST). T sends the ticket and a confirmation encrypted with K_CT, including C's nonce and S's identity.
  • Step 3: C sends the ticket to S, along with a nonce encrypted with the session key (authenticating C to S).
  • Step 4: S decrypts the ticket, obtains K_session, and responds with a nonce encrypted with K_session, including C's identity (authenticating S to C).
  • Security properties:
    • Mutual authentication: Step 3 and Step 4 provide mutual authentication through nonces.
    • Session key: K_session is generated by T and distributed securely.
    • Replay protection: Nonces ensure freshness.
    • MITM: Encryption with long-term keys prevents impersonation.
    • Reflection: The use of separate nonces and identities prevents reflection.
    • Forward secrecy: Not fully provided unless the session key is derived from ephemeral Diffie-Hellman (which would require additional messages).

Homework 3.6-3: Authentication Protocol Research Paper

Write a 1,500–2,000 word research paper on the evolution of authentication protocols. Cover at least three protocols from different eras (e.g., PAP/CHAP, Kerberos, EAP, TLS, and FIDO2). Your paper should address:

Sample Answer

Note: This is a research assignment. The sample answer provides an outline.

Evolution of Authentication Protocols

  • PAP and CHAP: Early protocols for point-to-point connections. PAP was insecure (cleartext). CHAP introduced challenge-response, but lacked mutual authentication.
  • Kerberos: Introduced a trusted third party, tickets, and mutual authentication. Established the foundation for enterprise single sign-on.
  • EAP and TLS: EAP provided a flexible framework; TLS added public-key cryptography and mutual authentication with certificates.
  • FIDO2: Modern passwordless authentication using public-key cryptography and phishing resistance.
  • Key themes: The evolution shows a progression toward stronger cryptographic guarantees, phishing resistance, and user convenience.

Homework 3.6-4: Protocol Implementation Review

Review the implementation of a real-world authentication protocol (e.g., TLS handshake, OAuth 2.0 flow, or SAML). Write a 750–1,000 word report that includes:

  1. A description of the protocol and its security goals.
  2. How the protocol protects against replay, MITM, and other attacks.
  3. Any known implementation vulnerabilities (e.g., Heartbleed, KRACK) and their root causes.
  4. How the protocol is used in a specific application (e.g., web authentication, wireless security).
  5. Recommendations for secure implementation and configuration.
Sample Answer

Review of the TLS 1.3 Handshake

  • Protocol description: TLS 1.3 establishes a secure connection between client and server, providing mutual authentication (with certificates), key establishment, and confidentiality.
  • Security features: Uses ephemeral Diffie-Hellman for forward secrecy; nonces for replay protection; authenticated encryption (AEAD) for integrity; and certificate verification for server authentication.
  • Known vulnerabilities: Heartbleed (2014) was an implementation flaw in OpenSSL that allowed memory exposure, not a protocol flaw. TLS 1.3 fixed many weaknesses of previous versions.
  • Application: TLS is used in HTTPS, secure email, and many other protocols.
  • Recommendations: Use TLS 1.3, properly validate certificates, use secure cipher suites, and keep libraries updated.

Homework 3.6-5: Authentication Protocol for IoT

Design an authentication protocol for a constrained IoT device (e.g., a smart sensor) that communicates with a cloud server. The device has limited processing power, memory, and battery life. It also has a hardware-based unique identifier. The protocol should provide mutual authentication and a session key, while minimizing computational and communication overhead. Consider using pre-shared keys, lightweight cryptography, and a trusted third party (e.g., a gateway). Provide a detailed design and justify your choices.

Sample Answer

Lightweight Authentication Protocol for IoT

  • Assumptions: Device and cloud server share a pre-shared key (K_DS). The device has a hardware identifier (ID_D).
  • Protocol: A lightweight challenge-response using AES-CCM (Authenticated Encryption with Associated Data) for efficiency.
  • Message flow:
    1. Device sends ID_D and a nonce (N_D) to the server.
    2. Server generates a nonce (N_S) and a session key (K_session). Server computes a challenge: C = AES-CCM(K_DS, N_D || N_S || K_session). Server sends N_S, C.
    3. Device decrypts C, verifies N_D, and extracts K_session. Device computes response: R = AES-CCM(K_session, N_S). Device sends R.
    4. Server verifies R and accepts the session.
  • Justification: AES-CCM is efficient and provides both confidentiality and integrity. The protocol uses minimal messages (3) and requires only a few cryptographic operations. The use of nonces provides replay protection.
  • Limitations: The pre-shared key must be securely provisioned. Key compromise would require re-provisioning.

Summary

This tutorial provided a comprehensive exploration of authentication protocols and mechanisms, which are essential for securing authentication over network communications. We began by defining authentication protocols and their desirable properties: freshness, mutual authentication, key establishment, and resistance to attacks.

We examined challenge-response authentication as a foundational technique that provides replay protection through the use of fresh, unpredictable challenges. CHAP was discussed as a classic example, and we compared it with more modern protocols like EAP-TLS. We also explored mutual authentication, emphasizing the need for both parties to verify each other's identities.

Replay protection mechanisms were analyzed in depth, including nonces, timestamps, and sequence numbers, each with its own trade-offs in terms of state management, clock synchronization, and security. We then covered session key establishment and tickets, which enable secure, efficient communication and single sign-on.

A significant portion of the tutorial was dedicated to protocol security analysis. We examined common attacks—replay, MITM, reflection, parallel session—and their countermeasures. The Needham-Schroeder vulnerability illustrated how subtle design flaws can lead to serious security weaknesses, and the WPA2 KRACK attack highlighted the importance of correct implementation and state management.

We surveyed common authentication protocols (PAP, CHAP, EAP) and compared their security properties and use cases. The case studies on Needham-Schroeder and the WPA2 4-way handshake provided real-world examples of protocol design and implementation challenges.

This tutorial has equipped you with the knowledge to analyze, design, and evaluate authentication protocols. In the next tutorial, Tutorial 3.7, we will dive into the Kerberos authentication architecture, a widely deployed protocol that leverages many of the concepts we have covered—challenge-response, mutual authentication, session keys, and tickets—to provide secure network authentication.

← Previous Tutorial (3.5) | Table of Contents | Unit Home | Next Tutorial (3.7) →

© 2026 COMP400 – Computer and Network Security • School of Computing and Information Systems, TrustOpen University • Unit 3: Authentication and Access Control