Intrusion detection systems (IDS) gained popularity in the 1990s to address weaknesses in computer software. Their detection strategy relies on signatures, which attempt to identify traffic patterns associated with known exploits of documented software vulnerabilities. When IDS detects a known exploit, it sends an alert to the security operations center (SOC) so analysts can investigate its severity.

In the early 2000s, IDS was considered the “source of truth” for the network—so much so that intrusion detection technology became a compliance requirement for the Payment Card Industry Data Security Standard (PCI DSS). Many cyber insurance issuers also require some form of IDS for coverage.

Over the roughly 25 years that IDS solutions have been a part of the security technology landscape, organizations relying on them have grown disillusioned with the technology. They’ve witnessed different attacks sneak past their IDS time and again.

As other security technologies have advanced, many IDS solutions lack the ability to learn. They only know the signatures and exploits included as part of their detection methodology. The threat landscape has evolved so much that decades-old signature detection only tells at best part of the story.

Machine Learning Strengthens Defense

From the beginning of IDS, searching for patterns, tracking behavior, and finding anomalies were required for full-spectrum detection (NIST 800-94). IDS developers understood early patterns, but detecting new behaviors and anomalies proved difficult to achieve in dynamic environments using manual analysis techniques.

Today, machine learning (ML) has become the foundation for delivering what is sorely lacking in threat detection and response solutions: behavioral, anomaly, peer group, and rules-based pattern detections. Machine learning is critically important because it allows the technology to detect both known and unknown attacker tactics, techniques, and procedures (TTP). Sophisticated threat actors will continuously find new and inventive ways to gain access to the network, and standalone IDS without network data has demonstrated it can’t keep up with both known and unknown threats.

Encrypted Data and an Expanding Perimeter

IDS was developed when most network traffic was delivered in cleartext. Today, encryption is the most common tactic for securing data. For example, 95% of Google traffic is encrypted. As a result, IDS is blind to much of the important, the mundane, and the dangerous traffic crossing the perimeter or moving laterally through data centers and cloud infrastructure.

The perimeter surrounding the network also continues to expand, with unmanaged devices and cloud workloads crossing boundaries without any observable security state. These devices can become infected outside of a security team's purview or through alternative communication channels, such as third-party VPN, mobile side channels, or trusted peering networks. These advanced methods are unobservable by outward-facing IDS.

Attacks are More Advanced

Software vulnerabilities were the primary threat consideration when designing IDS, and CVEs continue to be an important security concern. However, attackers now prefer advanced exploitation methods—through social engineering schemes, use of stolen credentials, or human error (misconfigurations)—finding these tactics more effective than swimming in assembly code to develop or buy expensive zero-day exploits.

The defender’s dilemma states that modern attackers only have to get it right once, whereas defenders have to get it right every time to prevent a breach at the perimeter. IDS lacks the defense-in-depth detection backup against attackers sneaking past legacy prevention defenses as they land and pivot toward valuable data.

Alert Fatigue

Legacy IDS inspects traffic as it passes by, looking for a pattern that matches a signature in its library. When it detects a match, IDS triggers an alert. Unfortunately, most IDS stops at alerts, leaving time-strapped analysts to search for root cause with other investigation tools and, in some cases, access another PCAP repository tool for forensic evidence.

These alerts have grown cumbersome for many SOC analysts. The more anomalies get detected, the more alerts are triggered. With a limited set of known detections, unknown signals will repeatedly sound the alarm, which can overwhelm analysts and increase the mean time to resolution.

The Next Evolution of IDS

There have been great improvements in cybersecurity, such as endpoint detection and response (EDR) and security information and event management (SIEM) solutions. However, these technologies still lack the broad visibility needed to improve the quality of alerts and eliminate blindspots.

One solution that can improve IDS functionality is network detection and response (NDR). NDR monitors both north-south and east-west traffic for malicious activity and policy violations. It also utilizes full-spectrum detection powered by ML behavioral analysis and high-risk CVE exploit identification, and combines those capabilities with streamlined incident response workflows.

When SOC analysts can see more, they know more clearly how to stop advanced threats. IDS is long overdue for an upgrade, and the next evolution should include key NDR capabilities to bolster a stronger security posture and maintain compliance requirements.