What is a Security Data Lake?

Table of Contents

An organization’s technology stack can generate terabytes or petabytes of data every day, including security data and information. As organizations seek to strengthen their security and compliance postures, they need modern technologies that enable them to gain the full value of all security telemetry, including unstructured data.

A security data lake often provides the technology foundation to a modern security data strategy and enables organizations to normalize and store all security and business data to leverage analytics models to support their security, privacy, risk and compliance strategies.

What is a security data lake?

A security data lake (SDL) is a centralized repository for storing security data, business data, policy context, and additional points of information that can together with security logs provide a broad and comprehensive picture for security analysis. This data can stored in a SDL in its raw, unstructured, or structured format. They are a scalable, cost-effective solution for storing and analyzing the large volumes of security telemetry that their environments generate every day, like log data or information from security analytics tools.A security data lake enables organizations to store and use:

Structured data: data from firewalls, anti-virus tools, endpoint detection and response (EDR) solutions
Unstructured data: threat intelligence feeds, dark web forum data, incident response reports
Semi-structured data: JSON, XML, NoSQL data formats

By reducing storage costs and enabling flexible, on-demand data processing, security data lakes enable organizations to gain the full value of their high volumes of security telemetry, like using it to automate threat hunting with advanced analytics models. Organizations often adopt a security data fabric on top of their SDL strategy to automate data engineering work required to correlate and connect data insights, accelerating analytical and operational use cases to more data consumers.

What are the benefits of a security lake?

For companies with complex, expansive IT and security environments, a security data lake offers benefits across operational, financial, and security domains.

Some fundamental benefits include:

Security data flexibility: ability to aggregate and correlate security data regardless of format
Centralized security data visibility: a single source of truth by normalizing all security telemetry collected from various tools and locations
Scalable data management: expand storage and compute on an as-needed basis to gain the full value of the security data
More affordable storage: decoupled, low-cost cloud storage for terabytes of petabytes of data providing comprehensive collection, aggregation, and analysis capabilities
Faster incident investigation: faster compute times across users so analysts can research incidents quickly, improving key metrics like Mean Time to Recover (MTTR)
Advanced security analytics: data scientists building behavior models and machine learning analytics projects, like dynamic dashboard displaying security metrics and risk indicators, for activities like threat hunting and automated response
Data enrichment: correlating event and non-event contextual data, like identity data (userID, host, IP address), vulnerability scans, and business context
High-fidelity detections: customized rules that model complex attacker behavior to reduce the number of alerts and false positives
Enhanced compliance: ability to create organization-specific key performance indicators (KPIs) and tracking dashboards with business intelligence (BI) tools for improved reporting
Cross-functional collaboration: centralized access to data across security, IT, compliance, and operational teams
No vendor lock-in: removing proprietary formats, like using the Open Cybersecurity Schema Framework (OCSF), for correlating all data across a diversified cybersecurity and IT technology stack

What is the difference between a security data lake and a security incident and event management (SIEM) tool?

A security data lake and a SIEM have different architectures because they solve different problems. A SIEM collects, correlates, and analyzes security events and creates workflows that enable rapid detection and response. The security data lake stores optimized and integrated data for storage.

Security data lakes differ from traditional SIEM solutions because security data is more available and can be retained for a longer time more cost effectively. This approach enables security teams to keep more of their raw data while breaking down security data silos for more comprehensive analysis. With greater flexibility and more comprehensive analysis of security data, organizations have more freedom to leverage machine learning and artificial intelligence-based analytics.

Data Storage Capabilities

A security data lake’s primary use case is to collect, normalize, aggregate, and store petabytes of security data in a cost-effective manner.

A SIEM was not intended as a data storage repository and costs can often become overwhelming, especially in cloud environments that generate petabytes of data.

Broad Detection Coverage

Using a security data lake, analysts can apply detection rules to a broader dataset that correlates business, risk, IT data, and security data. This is unlike a traditional SIEM approach where security analysts have security vendor-supplied detection rules and predefined use cases that they can customize by using the proprietary query language.

More Complete Investigations and Enhanced Threat Hunting

A security data lake that offers on-demand compute can store both raw and processed data for threat hunters to conduct multiple, simultaneous hunts to help accelerate investigations. With access to a broader dataset with greater retention, threat hunters can pinpoint incidents with greater accuracy, while enabling teams to scale their compute resources on demand for faster data processing and analysis. When more data is stored for more immediate querying, threat hunters aren’t worried about not having the data they need or focused on managing data infrastructure.

Whereas a SIEM requires event correlation so that security teams know what additional data sources to use when investigating an incident.

Data Control

With a security data lake, the company retains control over the data’s cloud storage so it can manage compliance with various international data protection laws, like the General Data Protection Regulation (GDPR).

With a SIEM, the company may need an on-premises or self-hosted deployment to maintain control over the data and infrastructure to comply with international data protection laws.