What is a Next Generation Network Intrusion Detection System?

Intrusion detection was first introduced to the commercial market two decades ago as SNORT and quickly became a key cybersecurity control.

Deployed behind a firewall at strategic points within the network, a Network Intrusion Detection System (NIDS) monitors traffic to and from all devices on the network for the purposes of identifying attacks (intrusions) that passed through the network firewall.

In its first incarnation, NIDS used misuse-based (rules and signatures) or anomaly-based (patterns) detection engines to analyze passing traffic and match the traffic to the library of known attacks. Once the attack was identified, an alert was sent to the security operations team.

While the technology continues to play a key role in the majority of enterprises, the Network Intrusion Detection System has fallen out of favor for two key reasons:

The rules-based engines used for detection were subsumed into the Next Generation Firewall (NG-FW), making it more cost effective for some organizations to deploy a unified capability;
Threat actors are adept at executing attacks that evade the signatures/rules/patterns used by both the traditional Network Intrusion Detection System as well as the unified NG-FW.

Machine Learning and the Rise of Next Generation Network Intrusion Detection Systems

Like a traditional NIDS, the function of a Next Generation IDS/IPS is to detect a wide variety of network-based attacks perpetrated by threat actors and contain these attacks, where feasible, using appropriate controls. Unlike a traditional NIDS, this technology leverages machine learning powered analytics engines that are capable of identifying attacks that evade traditional misuse-based and anomaly-based engines.

Network attack types that should be addressed by an NG-NIDS include:

Malware Attacks:
Malware is malicious software created to “infect” or harm a target system for any number of reasons. These reasons span simple credential access and data theft to data or system disruption or destruction. Today, an estimated 30% of malware in the wild is capable of evading traditional signature-based technologies. Most organizations addressed this through the deployment of endpoint detection and response (EDR) technologies. In relatively homogenous environments with firm control over the endpoint, endpoint controls may be adequate. For organizations with an array of client and server technologies, limitations over patch and update frequencies, IoT devices, or endusers over whom there is limited control, a strategically deployed NG-NIDS acts as a primary defense against “unknown” malware.

Below are common use cases wherein a Next Generation Network Intrusion Detection System can play a protective role:

Malicious websites – These attacks generally start at legitimate websites that have been breached and infected with malware. When visitors access the sites via web browser, the infected site delivers malware to the endpoint. Alternatively, doppelganger sites can be used to disguise malware as legitimate downloads.
Phishing/Spearphishing emails – Threat actors trick endusers into downloading attachments that turn out to be malware. Alternatively, threat actors trick users to click on a seemingly legitimate link to visit a website, from which malware is delivered.
Malvertising – In this case, threat actors use advertising networks to distribute malware. When clicked, ads redirect users to a malware-hosting website.

In each of these cases, the NG-NIDS sits between an internal user and external site and is capable of detecting malware and issuing a block request to a firewall or endpoint manager to contain the threat. In situations where the enterprise uses a split tunnel architecture or allows mobile workers to access the Internet without restriction, the NG-NIDS will see suspicious activity emanating from an infected device once reconnected to the corporate network. NOTE: While a NG-NIDS can be highly effective and easier to deploy than endpoint technologies, it is highly recommended that organizations use both. It is certainly the most effective means of protecting an organization with a mobile workforce.

Attacks that “Live off the Land”:
One of the more frightening and rapidly emerging categories of attack is known as a “living off the land”, fileless malware, or “in-memory” attack. These attacks are specifically created to start or complete an action that is untraceable by today’s security tools. Rather than downloading a file to a host’s computing device, the attack occurs in the host’s memory (RAM), leaving no artifact on disk. Powering down or rebooting an infected system removes all artifacts of the attack; only logs of legitimate processes running remain, thereby defeating forensic analysis.

How is this attack accomplished? The attacker injects malware code directly into a host’s memory by exploiting vulnerabilities in unpatched operating systems, browsers and associated programs (like Java, JavaScript, Flash and PDF readers). Often triggered by a phishing attack, the victim clicks on an attachment or a link to a malware-infected website or a compromised advertisement on a reputable site.

Once the malware is in memory, attackers can steal administrative credentials, attack network assets, or establish backdoor connections to remote command and control (C2) servers. Fileless attacks can also turn into more traditional file-based attacks by downloading and installing malicious programs directly to computer memory or to hidden directories on the host machine. The threat actor can also employ a variety of tactics to remain in control of the system after a shutdown or reboot.

The role of the NG-NIDS is to intercept suspicious machine code (usually in the form of obfuscated JavaScript, Powershell, or VB Script) and emulates how malware will behave when executed. Operating at line speeds, the NG-IDS determines what an input can do if executed and to what extent these behaviors might initiate a security breach. By covering all potential execution chains and focusing on malicious capacity rather than malicious behavior, the NG-NIDS vastly reduces the number of execution environments and the quantity of analytic results, thereby reducing the number of alerts that must be investigated.

Worms:

Worms are a form of self-propagating malware that does not require user interaction. WannaCry, for example, targeted a widespread Windows vulnerability to infect a machine. Once infected, the malware moved laterally, infecting other vulnerable hosts. Once the target is infected, any number of actions can be taken, such as holding the device for ransom, wiping user files or the OS, stealing credentials, or scanning the network for vulnerabilities.

A strategically deployed NG-NIDS, sitting in an internal network, is capable of detecting lateral spread of the worm and issuing a block request to a firewall or endpoint manager to contain the threat.

Web attacks:

In a web attack, public facing services – like web servers and database servers – are directly targeted for a variety of reasons: to deface the web server, to steal or otherwise manipulate data, or to create a launching pad for additional attacks. The most common means of attack in this category include:

Cross-Site Scripting (XSS) – An attacker injects malicious code into the web server which, in turn, is executed on an enduser’s browser as the page is loaded.
SQL Injection (SQLi) – An attacker enters SQL statements to trick the application into revealing, manipulating, or deleting its data.
Path Traversal – Here, threat actors custom craft HTTP requests that can circumvent existing access controls, thus allowing them to navigate to other files and directories.

An NG-NIDS, sitting behind the firewall and in front of a Web or database server is capable of detecting these attacks and issuing block requests to an application firewall.

Scan Attacks:

Scans are generally used as a means to gather reconnaissance. In this case, threat actors use a variety of tools to probe systems to better understand targets available and exploitable vulnerabilities.

An NG-NIDS, sitting behind the network firewall, is capable of detecting these probes and issuing block requests to the network firewall.

Brute force attacks:

The threat actor attempts to uncover the password for a system or service through trial and error. Because this form or attack takes time to execute, threat actors often use software to automate the password cracking attempts. These passwords can be used for any number of purposes, including modification of systems settings, data theft, financial crime, etc.

An NG-NIDS, sitting behind the network firewall and/or at strategic points within the network is capable of detecting brute force attacks and issuing block requests the network firewall.

Denial-of-service attacks:

Also known as distributed denial-of-service attacks (DDoS), DDoS attacks try to overwhelm their target – typically a website or DNS servers – with a flood of traffic. In this case, the goal is to slow or crash the system.

An NG-NIDS, sitting behind the network firewall, is capable of detecting DDoS attacks and issuing block requests to the network firewall.