After completing this tutorial, you should be able to:
In Tutorial 4.2, we explored the attack lifecycle, threat actors, and the methodologies adversaries use to compromise systems. Now we turn to the technologies that are designed to detect those intrusions: Intrusion Detection Systems (IDS). An IDS is a foundational component of security monitoring, providing the eyes and ears that alert defenders when malicious activity occurs.
This tutorial provides a comprehensive introduction to IDS fundamentals. We begin by defining intrusion detection and its objectives, then we examine the architecture of a typical IDS and its functional components. The core of the tutorial is dedicated to the three primary detection methodologies: signature-based (pattern matching), anomaly-based (behavioral profiling), and specification-based (rule-based protocol validation). We discuss how each works, their advantages and limitations, and real-world applications.
We also cover alerting and event analysis, including the concepts of false positives, false negatives, and the challenges of tuning detection systems. Finally, we consider IDS deployment architectures, sensor placement, and the integration of IDS into a security operations workflow. Two case studies illustrate how IDS can be effective (and where they can fail) in real incidents.
This tutorial sets the stage for the next tutorials on host-based IDS (HIDS) and network-based IDS (NIDS), which will delve into specific implementations and operational details. The content aligns with Stallings & Brown (2024), Chapter 8 and incorporates guidance from NIST SP 800-94 and industry best practices.
Intrusion Detection is the process of monitoring the events occurring in a computer system or network and analyzing them for signs of possible incidents, which are violations or imminent threats of violation of computer security policies, acceptable use policies, or standard security practices. An Intrusion Detection System (IDS) is the software or hardware that automates this monitoring and analysis process.
An IDS is a detective control, not a preventive one. Its primary purpose is to alert security personnel to potential malicious activity so that they can investigate and respond. In the context of the PDR/P²DR models, the IDS provides the Detection (D) capability. It does not block attacks directly (that is the role of Intrusion Prevention Systems, or IPS, which we will cover later), but it provides the visibility needed to trigger the Response (R) phase.
An Intrusion Prevention System (IPS) is essentially an IDS that is placed inline and has the ability to block or modify traffic to prevent an attack from succeeding. The distinction is important: IDS is passive (monitors and alerts), while IPS is active (monitors, alerts, and takes action). In this tutorial, we focus on the detection capabilities that are common to both.
The goal is to maximize TP and TN while minimizing FP and FN. However, there is often a trade-off: more sensitive detection may increase FPs, while less sensitivity may increase FNs. The acceptable balance depends on the organization's risk appetite and resources.
A typical IDS consists of the following functional components:
Figure 1: High-Level IDS Architecture
In many modern implementations, these components are integrated into a single platform or virtual appliance, but the functional separation remains.
Signature-based detection (also known as pattern matching or misuse detection) relies on a database of known attack patterns, called signatures. Each signature describes a specific attack in terms of byte sequences, packet headers, or other characteristics. The IDS compares incoming data (packets, log entries, etc.) against the signature database. If a match is found, an alert is generated.
Snort, a popular open-source NIDS, uses rules like this:
This rule alerts when a TCP packet from an external source to port 80 on the internal network contains the string "cmd.exe" – a classic indication of a web shell or command execution attempt.
Anomaly-based detection (also called behavior-based) establishes a baseline of "normal" activity for the system or network. Any activity that deviates significantly from the baseline is flagged as potential malicious. This approach does not rely on known attack signatures; instead, it identifies outliers.
Baselines can be created using statistical methods (e.g., mean and standard deviation of traffic volume), machine learning (e.g., clustering, neural networks), or heuristics.
An IDS might monitor the average outbound traffic per hour. If a host suddenly sends 10× more data than usual during off-hours, it may indicate data exfiltration. The IDS would generate an alert based on a statistical threshold.
Specification-based detection uses a set of rules that define the legitimate behavior of protocols, applications, or system components. These rules are typically derived from formal specifications (e.g., RFCs for protocols) or secure operation guidelines. If a system deviates from the specification, an alert is generated.
Unlike anomaly detection, which uses statistical baselines, specification-based detection uses predefined rules that are deterministic. It does not require a training period.
An IDS might check that a TCP packet's flags field complies with the TCP specification (e.g., a SYN packet must not have the ACK flag set). A packet that violates the spec could indicate a crafted attack packet.
In practice, most modern IDS/IPS solutions employ a hybrid approach, combining signature, anomaly, and specification-based methods to leverage the strengths of each. For example:
The outputs from multiple detection engines are correlated, and a scoring or prioritization mechanism is used to reduce false positives and highlight the most critical alerts.
When the IDS detects a suspicious event, it generates an alert. Alerts typically contain:
The raw alerts often require analysis to separate real threats from false positives. Security analysts review alerts, correlate with other data sources (e.g., SIEM), and determine appropriate response. Over time, the IDS configuration is tuned:
False positives are a major operational challenge. They waste analyst time and can lead to alert fatigue, where real threats are missed. False negatives are even more dangerous because they represent undetected attacks. The goal is to strike a balance that minimizes both, but achieving this requires continuous tuning and threat intelligence.
Attackers use various techniques to evade detection, such as:
Modern IDS must incorporate features like packet reassembly, decoding, and behavioral analysis to counter these evasions.
In the Target data breach, attackers gained access via a third-party HVAC vendor and moved laterally to compromise the point-of-sale systems. The company had deployed a NIDS, but the alerts were not acted upon in time. The IDS did generate alerts for suspicious activity, but they were ignored or deprioritized. This case highlights that detection is only as good as the response process.
Signature-based IDS were highly effective in detecting WannaCry, as the malware used a known EternalBlue exploit. Organizations that had up-to-date signatures and patch management were able to detect and block the attack. However, those without patched systems and updated signatures suffered widespread infections.
This tutorial provided a comprehensive introduction to Intrusion Detection Systems (IDS). We defined intrusion detection as the process of monitoring and analyzing events to identify signs of policy violations or attacks. We explored the architecture of a typical IDS, which consists of sensors, analyzers, and management consoles.
We examined three primary detection methodologies: signature-based, which matches known patterns and is accurate for known threats but blind to novel attacks; anomaly-based, which uses behavioral baselines to detect deviations, offering the ability to spot zero-days but suffering from higher false positive rates; and specification-based, which enforces protocol conformance and is precise but limited to well-specified components. We noted that hybrid systems combine these approaches to maximize detection coverage.
We discussed alerting mechanisms, the challenges of false positives and false negatives, and the importance of event analysis and tuning. We also covered IDS deployment considerations, including sensor placement, performance, and evasion techniques. Finally, we reviewed real-world case studies that illustrate both the successes and pitfalls of IDS implementation.
Next: In Tutorial 4.4, we will dive into Host-Based Intrusion Detection Systems (HIDS), exploring how they monitor individual hosts for suspicious activities, including file integrity, system calls, and logs.
1. Which of the following is a primary objective of intrusion detection?
2. Which detection method uses a database of known attack patterns?
3. What is the main advantage of anomaly-based detection over signature-based detection?
4. A false negative occurs when:
5. Which IDS component is responsible for collecting data from the monitored environment?
6. Specification-based detection relies on:
7. Which of the following is a common evasion technique used by attackers to bypass signature-based IDS?
8. In hybrid detection systems, which combination is common?
9. The primary challenge with anomaly-based detection is:
10. What is the main role of an IDS in the PDR model?
11. Which of the following is NOT a component of a typical IDS architecture?
12. A Snort rule that looks for the string "cmd.exe" in HTTP traffic is an example of which detection method?
Exercise 1: Signature Development Intermediate
You are tasked with creating a simple signature for a Snort-like IDS to detect a simple attack: an HTTP GET request containing the string "/etc/passwd". Write a rule (or describe the logic) that would alert on such traffic. Explain what fields you would match on, and discuss any potential false positives or evasion risks.
Rule logic: Alert on TCP traffic from any external IP to port 80 on the internal network, with content "/etc/passwd" in the HTTP request.
Snort-like rule: alert tcp $EXTERNAL_NET any -> $HOME_NET 80 (msg:"Possible /etc/passwd access"; content:"/etc/passwd"; http_uri; classtype:attempted-admin; sid:1001;)
Potential false positives: Legitimate requests for documentation or files named "/etc/passwd" on a web server. Evasion: URL encoding, case variations, or using POST instead of GET.
Exercise 2: Anomaly Baseline Design Intermediate
For a small corporate network, propose a set of metrics that an anomaly-based IDS could monitor to detect potential data exfiltration. Describe how you would establish a baseline and what thresholds might trigger alerts. Discuss how you would handle seasonal variations (e.g., end-of-month reporting).
Metrics: Outbound traffic volume per host (bytes/second), number of outbound connections per host, destination IP entropy (to detect connections to many unique IPs), protocol distribution (e.g., sudden spike in HTTPS or DNS traffic).
Baseline: Collect data over 4-6 weeks, calculate moving averages and standard deviations for each metric per host and time-of-day/week.
Thresholds: Alert when a metric exceeds 3 standard deviations above the mean for that time window. For seasonal variations, use time-series models that account for historical patterns (e.g., daily, weekly cycles).
Exercise 3: IDS Placement Analysis Advanced
Given the network diagram below (simplified), recommend where to place NIDS sensors to maximize detection coverage while minimizing resource usage. Justify each placement.
Recommended sensor placements:
This provides multi-layered visibility: external, DMZ, internal, and remote access.
Exercise 4: False Positive Analysis Introductory
Your IDS generates an alert: "Possible port scan from internal host 192.168.1.10 to external network." However, the host is a security scanner used by the IT team for vulnerability assessments. What actions would you take to reduce false positives without disabling detection for real port scans?
Actions:
Exercise 5: Hybrid Detection Strategy Advanced
Design a detection strategy that combines signature-based and anomaly-based methods to detect a multi-stage attack: an initial phishing email that downloads a backdoor, followed by C2 communication and later data exfiltration. Describe what each method would detect and how they could be correlated.
Signature-based: Detect known phishing attachments (e.g., macro-enabled documents with malicious signatures), known backdoor malware hashes, and known C2 domain/IPs from threat intelligence.
Anomaly-based: Detect behavioral deviations: a user receiving an unusual email with an attachment (flagged by UEBA), the host making new outbound connections to an IP never seen before (C2), or a sudden spike in outbound data volume (exfiltration).
Correlation: A SIEM can combine alerts: an email attachment alert from the email gateway, followed by a process execution alert from EDR, followed by an anomalous outbound connection alert from NIDS. The correlation increases confidence and provides a timeline for incident response.
Homework 1: Write a 1,500-word research paper comparing the effectiveness of signature-based vs. anomaly-based detection in cloud-native environments. Discuss the challenges of each approach when dealing with ephemeral workloads, containerized applications, and serverless architectures. Include at least 5 academic or industry references.
Key points: Cloud-native environments are dynamic, making baseline establishment difficult for anomaly-based detection. Signatures can be effective but may miss attacks that exploit cloud-specific misconfigurations. Hybrid approaches are recommended. References to NIST, Cloud Security Alliance, and recent academic papers.
Homework 2: Install and configure Snort (or Suricata) on a virtual machine or in a lab environment. Generate traffic (both normal and attack) and observe alerts. Write a report documenting your setup, rules you created or modified, and analysis of the alerts. Include screen captures of the alerts and a discussion of false positives/negatives.
This is a practical assignment; the report should include steps for installation, rule configuration, testing (e.g., using nmap, Metasploit, or custom scripts), and alert log analysis. Mention tuning adjustments made.
Homework 3: Research the evolution of IDS evasion techniques over the last decade. Write a 1,000-word essay that describes at least five evasion techniques (e.g., fragmentation, encryption, obfuscation, padding, spoofing) and how modern IDS/IPS systems have adapted to counter them. Include a discussion of the role of machine learning in identifying evasive patterns.
Outline: Introduction to evasion; fragmentation (overlapping fragments); encryption (TLS decryption challenges); obfuscation (encoding, polymorphism); padding (adding benign data); spoofing (IP/MAC). Countermeasures: deep packet inspection, stateful analysis, protocol dissection, anomaly detection for unusual patterns, and ML-based detection of evasion attempts.
Homework 4: Given the following network architecture, design a comprehensive IDS/IPS deployment plan. Include sensor types (NIDS/HIDS), placement, detection methods you would configure, and how you would integrate alerts with a SIEM. Justify your choices.
Plan: Place NIDS at HQ perimeter, internal segment boundaries, and in the cloud VPC. Use HIDS on critical servers and endpoints with EDR. Use signature-based for PCI requirements, anomaly-based for internal threats. Integrate with SIEM for correlation. Use inline IPS for critical segments (like payment network) to block known threats.
Homework 5: Write a critical analysis of the limitations of current IDS technologies in detecting Advanced Persistent Threats (APTs). Discuss why APTs often evade detection and propose enhancements (e.g., threat hunting, behavioral analytics, deception technologies) that can complement traditional IDS. Include a discussion of the role of human analysts.
Key arguments: APTs use low-and-slow tactics, legitimate credentials, and custom malware. Traditional signatures fail; anomaly detection may not flag subtle deviations. Enhancements: threat hunting with hypotheses based on intelligence, UEBA, network deception, and advanced analytics (e.g., graph analysis). Human analysts are crucial for interpreting context and performing deep investigations.
COMP400 – Computer and Network Security (Revision 3) • Unit 4: Security Systems and Models