Rolling your own Detections as Code with Elastic Security

From its beginning, the Elastic detection-rules repo not only contained Elastic’s prebuilt detection rules, but also additional tooling for detection rule management — like a suite of tests, CLI commands, and automation scripts used by the Elastic Threat Research and Detection Engineering (TRaDE) team.

Elastic’s TRaDE team has been following Detections as Code (DaC) practices for years, supporting our internal development and release processes for Elastic prebuilt detection and endpoint rules. Customers historically have predominantly used the detection-rules CLI, which provided a Kibana API wrapper and CLI commands to facilitate rule management with the Elastic SIEM, but it required users to adopt Elastic’s internal processes and rigorous restrictions.

With DaC becoming more mainstream and continuing with our commitment to openness, we are working on making it easier for users to kick-start their own DaC process using Elastic’s detection-rules repo for their rules management. This DaC expansion will build upon prior detection rules features to provide an end-to-end experience for detection engineers.

This methodology enhances collaboration among security teams, streamlines updates, and facilitates a more agile response to evolving threats. Fundamentally, we are committed to improving detection engineering maturity for our customers by exposing more internal capabilities for them to embed within their processes.

DaC adoption is driven by several factors:

• Drive toward security team maturity: Implementing DaC encourages the development of mature, repeatable processes within security teams. It supports the maintenance of high-quality detections through systematic peer reviews and rigorous testing.

• Ever-growing rule sets: Maintaining detection rules without DaC becomes untenable as the number of rules increases.

• Expanding threat landscape: The breadth of coverage needed to protect against emerging threats necessitates a scalable approach suitable for diverse skill sets.

• Broader adoption of automation: The wider movement toward automation in technology environments, exemplified by the adoption of IaC principles, calls for enhanced automation capabilities across all tools used by security teams. DaC aligns with this trend by integrating rule management into automated workflows, ensuring consistency, efficiency, and the ability to swiftly respond to new threats.

• Compliance and governance requirements: Many organizations are adopting DaC to meet certification compliance requirements for peer review, change control, and disaster recovery of SIEM detections. Implementing DaC ensures that security rules are developed, reviewed, and maintained following strict governance standards, providing auditable evidence of compliance.

Within the reference documentation and slides, you can read more about the core components of DaC: 

1. Maintaining rules within a Version Control System (VCS)

2. Syncing rules from VCS to their respective platform

3. Managing rules within the platform

4. Syncing rules from the platform to VCS

Additionally, we’ve documented different models to govern the core components of DaC using either:

1. VCS as authoritative

2. Platform as authoritative

3. Dual sync between VCS and the platform

Depending on the desired approach, we provide varying layers of detail, which becomes more abstract the further away from the Elastic Security solution.

Slides: These slides illustrate the high-level components, workflows, and delineations. It also touches on the benefits of using the detection-rules repo and a quickstart end-to-end reference example. 

Reference documentation: This reference document is packed with considerations, pros, and cons to developing, deploying, and managing detection rules, especially within Elastic Security environments. It is particularly beneficial for:

• Security analysts who wish to leverage automation for more efficient rule management and to respond more rapidly to emerging threats

• Detection engineers looking for methodologies to streamline detection logic development, testing, and deployment

• Security team leads seeking to implement best practices for rule version control, collaboration, and quality assurance within their teams

• DevOps engineers involved in integrating security practices into CI/CD pipelines, aiming for a more cohesive and automated approach to security within their workflows

• IT security architects exploring ways to incorporate as-code principles into security operations to enhance agility, repeatability, and reliability

For folks who would like a more pointed direction, we do provide a bias to the detection-rules approach. Note that the alpha detection-rules branch, the content within these slides, and this reference guide are subject to change. Once we finally migrate the changes to the `main` branch, we will update the content accordingly.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.