COMP 6660 · Enterprise Cybersecurity Network Security Protocols

00 Introduction

Lesson 4.2 established defence-in-depth as the principle of placing multiple, independent controls at different layers. This lesson examines the specific protocols through which the network layer of that defence is implemented in practice.

Network security protocols are the mechanisms through which enterprises protect data in transit, authenticate communicating parties, and verify the integrity of the network services on which everything else depends. They are the operational expression of the cryptographic foundations from Module 1: TLS, IPsec, SSH, and DNSSEC all build on the encryption, authentication, and integrity mechanisms examined in Unit 2. They are also the specific protocols that enterprise security programmes evaluate, configure, and monitor — where the abstract principles of architecture become concrete engineering decisions.

The protocols examined in this lesson are ubiquitous. Almost every enterprise connection — web traffic, API calls, remote administration, VPN connectivity, DNS resolution — depends on one or more of them. Their correct configuration determines whether the enterprise's traffic is genuinely protected or merely appears to be; their misconfiguration is a recurring source of enterprise exposure. And the legacy protocols that enterprises continue to carry — protocols that have been superseded but remain in use for compatibility — create risk that is frequently invisible until it is exploited.

This lesson examines network security protocols in detail. It establishes why protocol security matters and how protocols map to the layers of the OSI and TCP/IP models. It examines the four principal protocols — TLS, IPsec, SSH, and DNSSEC — their purposes, their mechanisms, and their configuration considerations. It examines other protocols relevant to specific contexts. It examines the legacy protocol risk that enterprises continue to carry. It examines protocol configuration and hardening as an operational discipline. And it closes with enterprise practice and the pitfalls that recur across organisations.

01 Learning objectives

By the end of this lesson you should be able to:

These objectives contribute to Unit 4 outcomes U4.1 (architecture design) and U4.2 (technology evaluation), and to Module 2 outcomes M2.5 and M2.6. See the Unit 4 learning outcomes page for the full mapping.

02 Why network protocols matter

Network security protocols are the operational mechanisms through which the enterprise's communication is protected. They matter for several reasons.

The evolution of network security

The protocols examined in this lesson have evolved substantially over decades. Early protocols — Telnet, FTP, HTTP — were designed for a trusted network environment and provided no confidentiality or integrity. Their successors — SSH, SFTP, HTTPS — layer security protocols on top of the original application protocols. This evolution is ongoing: TLS 1.0 and 1.1 have been deprecated; TLS 1.3 has replaced them. SSH has moved to modern key exchange and encryption. DNSSEC, though still incompletely deployed, is increasingly expected.

Enterprises that have kept pace with this evolution have substantially more secure communication than enterprises that have not. The gaps between them are frequently measured in decades of protocol development.

03 Protocol layers and security placement

Network security protocols operate at different layers of the network stack. Understanding which layer each protocol operates at clarifies what it protects and what it does not.

The layered network model

Where security protocols are placed

The placement of a security protocol determines what it protects and what it does not:

• Application-layer security. The protocol protects a specific application's traffic. SSH protects SSH sessions; S/MIME protects email; HTTPS protects web traffic. Application-layer protocols are specific to the applications they protect.
• Transport-layer security. The protocol protects communication between applications, independent of the application itself. TLS is the dominant example — it protects any application that uses it, without requiring the application to understand the protection.
• Network-layer security. The protocol protects traffic between hosts or networks, independent of the applications. IPsec is the dominant example — it protects all traffic between two endpoints, whether that traffic is web, email, file transfer, or anything else.
• Data-link-layer security. The protocol protects traffic on a specific link — Wi-Fi, Ethernet. These protocols protect against interception or modification on the local network.

Implications of placement

The placement of security protocols has several implications:

• Coverage. A network-layer protocol protects everything that traverses it, but it cannot distinguish between different applications. An application-layer protocol protects only the specific application.
• Visibility. Network-layer encryption hides traffic from any intermediate inspection — including security monitoring. This is why IPsec deployment requires careful consideration of what inspection is lost.
• Complexity. Application-layer protocols are typically simpler to deploy and configure; network-layer protocols are more complex but provide broader coverage.
• Overlap. Multiple protocols may be used in the same connection — TLS for the web session, IPsec for the tunnel carrying that session, WPA3 for the Wi-Fi link. Each layer provides its own protection.

04 TLS — Transport Layer Security

What TLS provides

• Confidentiality. Traffic between client and server is encrypted, preventing interception and reading.
• Integrity. Traffic is protected against modification in transit; the recipient can detect whether the content has been altered.
• Authenticity. The server (and optionally the client) is authenticated through digital certificates, providing assurance of the parties' identity.
• Forward secrecy. In modern configurations, TLS provides forward secrecy — compromise of a long-term key does not compromise past sessions.

How TLS works (conceptually)

05 IPsec — Internet Protocol Security

What IPsec provides

• Confidentiality. Traffic is encrypted at the network layer, protecting it from interception.
• Integrity. Traffic is protected against modification; the recipient can verify that it has not been altered.
• Authenticity. Communicating parties are authenticated through certificates, pre-shared keys, or other mechanisms.
• Replay protection. Duplicate packets are detected and discarded.
• Coverage across applications. Because IPsec operates at the network layer, it protects all traffic between the endpoints — regardless of which applications are involved.

IPsec components

IPsec applications

IPsec is used in several enterprise contexts:

• Site-to-site VPNs. Connecting enterprise networks — offices, data centres, cloud environments — across public networks.
• Remote access VPNs. Connecting remote users to the enterprise network — though SSL VPNs and zero-trust access increasingly compete with IPsec for this application.
• Cloud connectivity. Connecting enterprise networks to cloud environments through IPsec tunnels.
• Host-to-host security. Protecting communication between specific hosts — less common than in the past but still used in some contexts.

IPsec configuration considerations

• IKE version. Use IKEv2 in preference to IKEv1. IKEv2 provides better security, performance, and feature support.
• Authentication method. Prefer certificate-based authentication to pre-shared keys. Pre-shared keys are easier to configure but harder to manage at scale and frequently shared across many endpoints.
• Encryption algorithms. Use modern algorithms — AES-GCM, AES-CBC with HMAC-SHA-2. Avoid DES, 3DES, and weak cipher suites.
• Key exchange. Use Diffie-Hellman with adequate group sizes or elliptic-curve Diffie-Hellman (ECDH).
• Perfect forward secrecy (PFS). Enable PFS in the IKE configuration to ensure that compromise of a key does not compromise past or future sessions.
• Lifetime parameters. Configure reasonable lifetimes for IKE security associations and IPsec security associations.
• Dead peer detection. Enable detection of failed peers so that tunnels are re-established promptly.

IPsec deployment challenges

• Inspection visibility. IPsec encryption hides traffic from network inspection — including security monitoring. The enterprise loses visibility into the traffic within the tunnel.
• NAT traversal. IPsec traffic frequently conflicts with NAT, requiring NAT-Traversal (NAT-T) configuration.
• Interoperability. Different IPsec implementations may have subtle incompatibilities, particularly with configuration parameters and extensions.
• Complexity. IPsec configuration is substantially more complex than TLS configuration; misconfigurations are more common and harder to diagnose.
• Performance. IPsec encryption adds processing overhead, particularly for high-throughput tunnels.
• Monitoring. Tunnel status, throughput, and errors require specific monitoring, distinct from the monitoring of unencrypted traffic.

06 SSH — Secure Shell

What SSH provides

• Encrypted remote access. Interactive sessions and command execution are encrypted, protecting credentials and commands from interception.
• Server authentication. The client verifies the server's identity through host keys, protecting against man-in-the-middle attacks.
• Client authentication. The server verifies the client's identity through passwords, keys, or certificates.
• Secure file transfer. SFTP and SCP provide encrypted file transfer over SSH.
• Tunnelling. SSH can tunnel other protocols, providing a secure channel for otherwise-insecure traffic.
• Port forwarding. SSH can forward ports locally or remotely, enabling access to otherwise-inaccessible services.

SSH authentication methods

SSH configuration considerations

• Disable password authentication. Where feasible, require key-based or certificate-based authentication. This eliminates the most common SSH attack — credential guessing.
• Disable root login. Require administrators to authenticate as themselves and use privilege escalation (sudo) rather than logging in directly as root.
• Restrict access by user and source. Limit which users can connect, and from which networks, to reduce exposure.
• Use modern key exchange and encryption algorithms. Disable outdated algorithms (SSHv1, DSA, SHA-1, weak ciphers) and configure modern algorithms (Curve25519, Ed25519, AES-GCM).
• Host key management. Manage host keys securely; establish a mechanism for host key verification that resists man-in-the-middle attacks.
• Session timeout and inactivity limits. Configure timeouts to limit the window during which an abandoned session remains accessible.
• Logging and monitoring. Log SSH activity — successful and failed authentication, command execution, session duration — and monitor for anomalies.
• Privileged access management. Integrate SSH access with PAM controls — vaulting, session recording, just-in-time access — for administrative access to production systems.

Common SSH risks

• Exposed SSH ports. SSH services exposed to the internet, providing an attack surface for brute-force and exploitation attempts.
• Weak or shared credentials. Passwords that can be guessed; keys shared across administrators or across systems.
• Unmanaged authorized_keys files. SSH public keys accumulated over time in authorized_keys files, including keys for departed employees or unused systems.
• Disabled host key verification. Clients configured to ignore host key changes, allowing man-in-the-middle attacks.
• Weak key types. SSH keys using weak algorithms — DSA, RSA keys below 2048 bits, keys with weak passphrases.
• Exposed private keys. SSH private keys stored without passphrases, in shared locations, or committed to code repositories.

07 DNSSEC and DNS security

Why DNS security matters

DNS is the address book of the internet, and of the enterprise's internal networks. Every connection begins with a DNS resolution — a query that maps a name to an address. If that resolution is manipulated, the connection goes somewhere else — to an attacker's server, or to a service that the attacker controls.

DNS was designed in an era when the internet was trusted, and it lacks the authentication mechanisms that modern security requires. The result is that DNS is a target for several classes of attack:

• Cache poisoning. The attacker injects false DNS records into a resolver's cache, redirecting subsequent queries to attacker-controlled addresses.
• DNS spoofing. The attacker responds to DNS queries with forged responses, redirecting the querying system.
• DNS hijacking. The attacker compromises the DNS configuration of a domain, redirecting traffic.
• DNS tunnelling. The attacker uses DNS queries and responses as a covert channel for command-and-control or exfiltration.
• DDoS via DNS. DNS is used for amplification attacks — the attacker sends queries with spoofed source addresses, and the responses are sent to the victim.

What DNSSEC provides

DNSSEC provides cryptographic authentication of DNS responses. Authoritative DNS servers sign their responses; resolvers verify the signatures using a chain of trust that begins at the DNS root. This protects against cache poisoning and DNS spoofing — because forged responses cannot be signed with the domain's key.

DNSSEC does not provide confidentiality — DNS queries and responses remain readable. It does not protect against all DNS attacks — DNS tunnelling and DDoS are not addressed by DNSSEC. And it does not protect against compromise of the DNS server itself, if the attacker obtains the signing keys.

DNSSEC deployment

DNSSEC deployment has been slow, for several reasons:

• Complexity. DNSSEC configuration is more complex than standard DNS, and errors can produce resolution failures.
• Key management. DNSSEC requires cryptographic key management — signing keys, key rotation, key rollover procedures — that adds operational burden.
• Incomplete adoption. DNSSEC provides protection only when the entire chain from root to domain is signed and validated. Domains that do not sign, or resolvers that do not validate, break the chain.
• Breakage risk. Misconfigured DNSSEC can produce resolution failures that are difficult to diagnose, discouraging adoption.

Despite these challenges, DNSSEC adoption is increasing, particularly for government domains, financial services, and other high-value targets. The trend is clear: DNSSEC is becoming a baseline expectation for enterprise domains.

Other DNS security mechanisms

Beyond DNSSEC, several mechanisms improve DNS security:

08 Other security protocols

Beyond the four principal protocols, several other security protocols are relevant to enterprise environments.

Application-specific security protocols

Link-layer security protocols

Modern application protocols

09 Legacy protocol risk

Enterprises carry legacy protocols — protocols that have been superseded but remain in use. The reasons for their persistence are understandable: compatibility with legacy systems, operational convenience, cost of migration. The consequences are frequently underestimated.

Common legacy protocols and their risks

Why legacy protocols persist

• Legacy systems. Systems that do not support modern protocols — old devices, old applications, old operating systems.
• Compatibility requirements. Interoperability with partners, suppliers, or customers who use legacy protocols.
• Operational convenience. The effort of migrating to modern protocols, and the perceived risk of disruption.
• Visibility gaps. Enterprises frequently do not know which legacy protocols are in use — the protocols are embedded in systems, configurations, and integrations that have not been reviewed for years.
• Lack of ownership. Legacy protocols may not have a clear owner — they persist because no one is accountable for their remediation.

Managing legacy protocol risk

Managing legacy protocol risk requires a structured approach:

1. Inventory Identify which protocols are in use across the enterprise estate — including in systems, applications, integrations, and configurations that have not been reviewed recently.
2. Assess For each protocol, assess the risk — what data does it protect, who has access to it, and what is the consequence if it is exploited?
3. Prioritise Prioritise remediation by risk — protocols protecting sensitive data or providing access to critical systems warrant the highest priority.
4. Remediate Migrate to modern protocols where feasible. Where migration is not feasible, apply compensating controls — network segmentation, monitoring, access restrictions.
5. Monitor Monitor for legacy protocol use — both to detect unauthorised use and to track progress of remediation.
6. Review Review the protocol portfolio periodically. The threat landscape evolves; protocols that were acceptable yesterday may not be acceptable today.

10 Protocol configuration and hardening

Configuring protocols correctly is an operational discipline. The difference between a secure protocol configuration and an insecure one is frequently a small number of settings — but those settings determine whether the protocol provides the protection it is capable of providing.

Configuration principles

Hardening benchmarks

Several organisations publish hardening benchmarks for protocols and systems:

• CIS (Center for Internet Security) Benchmarks. Detailed configuration guidance for operating systems, applications, and network devices, including protocol configuration.
• NIST Guidelines. NIST publishes guidance on TLS configuration, IPsec configuration, SSH configuration, and DNSSEC deployment.
• Vendor hardening guides. Most major vendors publish hardening guidance for their products, including recommended protocol configurations.
• Mozilla SSL Configuration Generator. A widely used tool for generating modern TLS configurations for common server software.
• SSL Labs / testssl.sh. Tools that assess TLS configurations against industry best practice.

Continuous assessment

Protocol configuration is not a one-time activity. Protocols evolve, vulnerabilities are discovered, and configurations drift over time. Enterprises should assess their protocol configurations continuously — through automated scanning, periodic review, and integration with vulnerability management.

11 Enterprise practice and pitfalls

Enterprise network protocol programmes fail in predictable ways. Recognising the pitfalls accelerates improvement.

Pitfall 1 — Configuration without assessment

Configuring protocols with default settings, or with settings chosen for compatibility rather than security, without assessing what the configuration actually provides.

Mitigation: Assess protocol configurations against published guidance (CIS, NIST, vendor). Test configurations with tools like SSL Labs or testssl.sh.

Pitfall 2 — Legacy support without justification

Retaining deprecated protocol versions, algorithms, and cipher suites for compatibility without examining whether the compatibility is necessary or what it exposes.

Mitigation: Identify what requires legacy support; assess whether the requirement can be addressed through client upgrades, isolation, or compensating controls. Justify each exception explicitly.

Pitfall 3 — Unmanaged certificates

Certificates that are unmanaged — no inventory, no monitoring for expiry, no process for renewal — producing outages and creating exposure.

Mitigation: Maintain certificate inventory; monitor for expiry; automate renewal where feasible (ACME and similar protocols).

Pitfall 4 — Disabled validation

Applications and clients configured to skip certificate validation, ignore host key changes, or accept any certificate — undermining the protection regardless of server configuration.

Mitigation: Audit client configurations for validation bypasses. Address the underlying reasons for the bypasses (often certificate management issues) rather than allowing the bypasses to persist.

Pitfall 5 — Unmanaged SSH keys

SSH public keys accumulated in authorized_keys files without management — keys for departed employees, retired systems, and forgotten automation, each providing continuing access.

Mitigation: Inventory SSH keys; review authorized_keys files periodically; remove keys that are no longer needed; prefer certificate-based authentication over long-lived keys.

Pitfall 6 — DNSSEC without operational readiness

Deploying DNSSEC without the operational capability to manage the keys correctly — producing outages, breakage, and a reputation for DNSSEC as problematic.

Mitigation: Ensure key management capability before deploying DNSSEC. Use established tooling. Test carefully. Monitor validation.

Pitfall 7 — No protocol inventory

Not knowing which protocols are in use across the enterprise estate — producing invisible risk and preventing effective management.

Mitigation: Build and maintain a protocol inventory. Scan for protocol use; review configurations; track what is in use and where.

Pitfall 8 — No continuous assessment

Treating protocol configuration as a one-time activity — without monitoring for drift, without responding to new vulnerabilities, without updating for new best practice.

Mitigation: Establish continuous assessment for protocol configuration. Integrate with vulnerability management. Review periodically.

12 Key concepts

The terms below are used throughout Unit 4. Capture them in your own words as you work through the lesson.

TLS (Transport Layer Security)

The dominant protocol for protecting data in transit. Provides confidentiality, integrity, and authenticity for application communication, primarily over TCP.

Forward secrecy

The property that compromise of a long-term private key does not allow decryption of past sessions. Provided by ephemeral key agreement (ECDHE in TLS, PFS in IPsec).

HSTS (HTTP Strict Transport Security)

A mechanism that instructs browsers to use HTTPS for all connections to a domain, preventing downgrade attacks and cookie hijacking.

IPsec (Internet Protocol Security)

A suite of protocols for securing IP communications at the network layer. Provides confidentiality, integrity, authenticity, and replay protection independent of applications.

IKE (Internet Key Exchange)

The protocol through which IPsec peers negotiate security associations — algorithms, keys, and parameters. IKEv2 is the modern version.

Tunnel mode / Transport mode

IPsec modes. Tunnel mode encapsulates the entire IP packet and is used for VPNs; transport mode protects only the payload and is used for host-to-host communication.

Perfect Forward Secrecy (PFS)

The IPsec equivalent of forward secrecy — ensures that compromise of a key does not compromise past or future sessions.

SSH (Secure Shell)

A protocol for secure remote administration and secure file transfer. Provides encrypted and authenticated communication between client and server.

authorized_keys

The file on an SSH server that lists public keys authorised to authenticate. Frequently accumulates unmanaged keys over time.

DNSSEC (Domain Name System Security Extensions)

A set of extensions to DNS that provide cryptographic authentication of DNS responses — protecting against cache poisoning and spoofing.

DNS over HTTPS (DoH) / DNS over TLS (DoT)

Protocols that encrypt DNS queries between client and resolver, protecting against observation and manipulation on the local network.

Cache poisoning

An attack in which the adversary injects false DNS records into a resolver's cache, redirecting subsequent queries to attacker-controlled addresses. DNSSEC protects against this.

Legacy protocol

A protocol that has been superseded by a more secure version but remains in use for compatibility. Legacy protocols are a recurring source of enterprise exposure.

Cipher suite

A set of algorithms for key exchange, authentication, encryption, and integrity used by a security protocol. Cipher suites are negotiated during connection establishment.

Protocol version

The specific version of a protocol in use. Protocol versions are deprecated as weaknesses are discovered; deprecated versions should be disabled.

Hardening benchmark

Published guidance on secure configuration for systems, applications, or protocols — typically from CIS, NIST, or vendors.

Configuration drift

The tendency for configurations to change over time from the intended state — through manual changes, updates, or neglect. Continuous assessment detects drift.

13 Activities

The activities below help you test your understanding and prepare for the unit discussion and Assignment 2. They are not graded, but completing them will substantially improve your preparation.

14 Self-check

Answer the questions below before moving to Lesson 4.4. If you cannot answer a question confidently, revisit the corresponding section.

15 Summary

This lesson has examined network security protocols — the specific mechanisms through which the network layer of defence-in-depth is implemented. The essential points are:

• Network security protocols protect data in transit, establish trust between parties, and are ubiquitous in enterprise environments. Their correct configuration determines the enterprise's exposure across a wide range of attack vectors.
• Protocols operate at different layers of the network stack. Application-layer protocols (SSH, S/MIME), transport-layer protocols (TLS), network-layer protocols (IPsec), and link-layer protocols (WPA3, MACsec). Placement determines what each protects.
• TLS is the dominant protocol for protecting data in transit. Provides confidentiality, integrity, and authenticity; forward secrecy in modern configurations. TLS 1.2 and 1.3 are the modern versions; earlier versions are deprecated. Configuration considerations include protocol versions, cipher suites, certificates, and client validation.
• IPsec secures IP communications at the network layer. Used for site-to-site VPNs and cloud connectivity. Configuration considerations include IKE version, authentication method, algorithms, and forward secrecy. The principal limitation is that encryption hides traffic from inspection.
• SSH is the dominant protocol for remote administration. Configuration considerations include disabling password authentication, disabling root login, restricting access, and managing host keys and authorized_keys. The most consequential risk is the accumulation of unmanaged keys.
• DNSSEC provides cryptographic authentication of DNS responses. Protects against cache poisoning and spoofing. Deployment has been slow due to complexity and key management requirements, but is increasing. Other DNS mechanisms include DoH, DoT, filtering, and monitoring.
• Legacy protocols are a recurring source of enterprise exposure. Telnet, FTP, HTTP, SMBv1, SSL/TLS 1.0/1.1, NTLMv1, MD5/SHA-1. They persist for understandable reasons; the consequences are frequently underestimated.
• Protocol configuration is an operational discipline. Modern versions and algorithms; forward secrecy; strong authentication; client-side validation; visibility and monitoring; centralised configuration management. Hardening benchmarks (CIS, NIST) provide guidance.
• Common pitfalls recur. Configuration without assessment; legacy support without justification; unmanaged certificates; disabled validation; unmanaged SSH keys; DNSSEC without operational readiness; no protocol inventory; no continuous assessment.

These ideas prepare for Lesson 4.4, which examines virtual private networks — the mechanisms through which enterprises extend their networks across public infrastructure using protocols such as IPsec and TLS.