Elastic’s detection rules are still open for business — plus further visibility to data quality

Given the increasing frequency, tempo, and sophistication of cyber attacks, it's more important than ever that organizations take a proactive approach to protecting their sensitive data and assets. One of the most critical elements of effective cybersecurity is openness. An open approach to cybersecurity means being transparent, sharing knowledge, and collaborating to build more robust defenses against threats.

It's common practice in the cybersecurity market to provide customers with a “black box” tool and ask them to trust its capabilities to protect them. In contrast to other security tools, Elastic Security takes an open and transparent approach, which allows you to peek behind the scenes, understand how our instrumentation works, collaborate with us, and influence the product development process.

In this article, you’ll learn about our open detection content, how our open schema works with hundreds of diverse data sources, how we’re providing more visibility to data quality, and the critical role data quality plays in creating effective security capabilities.

Open detection content

Since the earliest days of Elastic Security, we have taken an open approach with the detection rules by developing them in the open and sharing them with the community via a public GitHub repository. We give our users transparency about the technology that protects their enterprise. Last year we also opened our endpoint artifacts, so everyone can see for themselves what kind of rules are running on Elastic Security for Endpoint. 

The Elastic Security 8.8 release comes with about 800 SIEM detection rules, tuning the rules and extending the coverage with a few more MITRE ATT&CK (R) techniques and subtechniques, as we do every release. We also provide over 380 Endpoint malicious behavior prevention rules, further expanding the coverage on endpoints. 

How do our rules automatically work with endpoint data, firewall data, cloud data, and more?

This is where Elastic Common Schema (ECS) comes into play. ECS is a community-driven, open-source specification that standardizes and normalizes fields and data types used in logs and events with our security solution.

An open schema for cybersecurity: Elastic Common Schema

The Elastic Common Schema (ECS) is an open-source schema that defines a common set of fields and data types for logs and metrics in Elasticsearch. ECS provides a consistent and interoperable way to organize data, making it easier for security teams to analyze and respond to threats by correlating information from a variety of different sources. Because ECS is also developed with our community, it evolves and adapts to new threats and vulnerabilities more quickly than proprietary (closed) solutions, allowing for greater flexibility and innovation in cybersecurity.

We welcome more participants to contribute to ECS. ECS allows you to benefit from Elastic’s prebuilt detection content, allowing those resources to work with your environment and data. It also enables Elastic to provide you with future turn-key capabilities that work out of the box.

In April 2023, Elastic contributed ECS to Open Telemetry and committed to joint development of a common schema. OTel is the second highest velocity project in the Cloud Native Computing Foundation (CNCF) and provides a collection of tools, APIs, and SDKs used to generate, collect, process, and export telemetry data (metrics, logs, and traces) for understanding software performance and behavior. Elastic has long been a proponent of open standards, and this contribution continues and expands Elastic’s commitment to the community. By contributing ECS to OpenTelemetry, we are hoping to create a mature and proven common schema in OTel for metrics, logs, traces, and security events based on the widely deployed Elastic Common Schema. Together with OTel, we will continue to develop and support that common schema going forward.

Better data, better protections: Data Quality Dashboard

In today's data-driven world, maintaining high-quality data is crucial for effective security measures and reliable insights. We are excited to unveil the latest addition to Elastic Security with our 8.8 release: the Data Quality Dashboard. 

This powerful tool is designed to provide users with a comprehensive solution for assessing and monitoring data quality within their environment. With the Data Quality Dashboard, you can easily ensure that all your data is correctly mapped to Elastic Common Schema (ECS), giving you confidence in the accuracy and reliability of your data. In this section, we will delve into the functionality and benefits of the Data Quality Dashboard, highlighting its ability to help you quickly identify and resolve data quality issues, thus enhancing the accuracy, completeness, and timeliness of your data.

ECS is a permissive schema, which means that you can easily bring custom data sets into Elastic Security. In the past, users lacked a robust method to evaluate data quality in their Elastic Security environment. However, with the introduction of the Data Quality Dashboard, that has changed. With just a single click, you can now check all your indices for any mapping conflicts and effortlessly share the results with other members of your organization, fostering collaboration and transparency.

The Data Quality Dashboard offers a holistic view of your data in Elastic. You can easily find the size of each index and index pattern between your different ILM tiers. Additionally, a summary of incompatible field mappings within each index allows you to quickly identify fields where data is mapped incorrectly or incompatible field values exist. This level of detail provides you with the necessary insights to better understand your data and address data quality issues promptly. By resolving these issues, you can ensure that your data remains accurate, complete, and up to date.

Maintaining accurate, timely, and reliable data is paramount in detecting and preventing cyber threats effectively. By ensuring that all data is accurately mapped to ECS, the Data Quality Dashboard acts as a crucial line of defense. Using this dashboard to identify and resolve data quality issues, organizations can bolster their cybersecurity measures, protecting sensitive data and assets from potential breaches.

Explore more

As part of our commitment to openness and continuous improvement, we encourage you to explore our latest 8.8 release and try out Elastic Security on Elastic Cloud. Your feedback and ideas are invaluable to us, so please don't hesitate to reach out to us on our community Slack or post on the Discuss forum. We look forward to hearing from you and stay tuned for more exciting updates!