Know your tools: The full range of Elastic Security’s detection engineering capabilities

Whether you’re a dedicated detection engineer or you wear multiple hats, welcome! Thanks for stopping by to read about the tools that Elastic Security has for you.

First, let's briefly go through the new capabilities added in Elastic Security 8.16.

• Alert suppression for Elasticsearch Query Language (ES|QL), machine learning (ML), threshold, indicator match, and new terms rule types are now fully supported and generally available. With suppression, you can reduce the volume of similar alerts (per rule run or window of time), resulting in decreased alert fatigue and time-efficient alert triage. Suppression capabilities require a Platinum or higher license tier.

• With manual rule runs, you can now test your rule or rerun it over the selected period up to 90 days in the past, which helps detection engineers assess the quality and noise level of the newly designed rule using historical events. This functionality is available at the Standard tier and is in beta as we continue to develop additional features.
• Security teams can now automatically create a case for Elastic Security alerts to streamline investigations with aggregation capabilities that combine multiple alerts into a single case. It is currently available in technical preview.
• There is a new option to enable prebuilt rules at the time of installation to make this process more smooth.
• Enhanced rule preview for EQL and ES|QL rules with the option to view Elasticsearch requests that will be executed.

Now that we’ve discussed the specifics of 8.16, we’ll dive deeper into all of the threat detection capabilities.

If you just started using Elastic Security, we got you covered with initial detection rules selection.

At the time of writing this blog (v8.16), we have more than 1,230 out-of-the-box SIEM detection rules across 54 different data sources and over 70 machine learning jobs — both trained models and anomaly detection jobs — to get started.

You can choose which rules to install and preview their content to understand the logic and additional details.

With 8.16, when you are installing the rule, you can immediately enable it at the time of installation.

Rules shipped by Elastic provide additional context explaining the rule prerequisites and providing advice on alert investigation — users can check the required data source integrations, setup, and investigation guides for this information. Our rules are mapped to the relevant MITRE ATT&CK tactics, techniques, and subtechniques, where those can be clearly defined.

With Elastic Security’s prebuilt rules, we continuously research threats, update and tune existing rules, and add new ones. We listen to our community’s feedback and constantly look for the false positive reduction possibilities and performance improvements, which are reflected in the rule query updates.

With biweekly updates, you can always see what exactly has changed in the rule in a convenient side-by-side view as shown below.

If you have suggestions or feedback or want to contribute, you can always open an issue in the public detection rules repository and follow our development process there.

Using detection rules, you will notice that some rules mark alerts as building blocks, which means they are not meant for triage and/or investigation and will not show up in the default Alerts view.

Building block rules (BBR) are there to elevate atomic activity for threat-hunting purposes and influence risk scores of entities. You can build more robust rules on top of such building block alerts. 

If you are interested in more threat-hunting use cases, read this article and visit the threat hunting folder of the detection rules repository to check out the hunting library.

You can write your detection logic in ES|QL, Kibana Query Language (KQL), Lucene, or EQL.

The ES|QL rule type allows you to write a very flexible detection logic, passing data from one part of the query to the other and manipulating it in the query itself. ES|QL is the newest of Elastic query languages and is in active development, so keep an eye out for new capabilities that will be useful in detection use cases.

The custom query rule type is useful for the single event match — queries can be written in KQL or Lucene.

The event correlation rule type is great for event sequence detection and can be written in EQL. You can also use this rule type to detect if an event is missing in a sequence.

Furthermore, Elastic AI Assistant can help create a rule query from scratch if given the specific use case you want to detect.

Elastic AI Assistant can be used in all security workflows and is especially helpful in alert analysis, streamlining workflows, and automating triage and remediation. 

Keep an eye on developments in this and other generative AI capabilities in Elastic Security as they become more and more powerful.

• Use the threshold rule type to alert when the volume of events and cardinality of a field exceeds the threshold, such as multiple failed logins from the same username and 10 different source IP addresses.

• New terms rules find a new term that was not seen before in the historical time window, such as successful authentication from a new user to a critical server.
• Indicator match helps detect matches between incoming threat intelligence and logs or logs with the predetermined list of indicators. For example, a user requesting access to a known malicious domain or url (a known bad) can be an indicator of compromise.

You can also write rules based on the BBR alerts from other so-called higher order rules. An example of a higher order rule is the detection of an alert sequence that indicates an attack chain and spans multiple tactics and techniques or if multiple building block alerts are triggered for the same user/host/IP, indicating a highly suspicious activity.

While working on the rule, it is important to check if it behaves as expected! You can do this using preview functionality.

Alert noise can be reduced by using alert suppression. When tuning for suppression, you can select up to three fields to suppress by and choose whether to suppress alerts per each rule execution or for a specified period of time.

During rule execution, alerts with matching suppression fields are grouped and only one alert is created — this includes the counter of grouped alerts. Analysts can see the number of detections with the same suppression fields, the suppressed values, and the suppression start and end time range but are not flooded with a huge number of duplicate alerts.

If you need to view the original events for the suppressed alerts, you can copy and run the rule query for a needed time frame in Discover or Timeline.

To further streamline analysis, you can set up custom highlighted fields.

Knowing now what detections Elastic Security provides out of the box and threat detection capabilities you can use to create custom detections, you also need to plan your detection coverage and focus efforts on threats that matter to your organization.

You can quickly enable all rules for a specific technique that you've installed but not yet activated right from the technique cell you are looking into.

As part of standard practices, detection engineering teams should regularly review particularly noisy rules or rules that are unusually quiet.

There are a few ways of addressing false positives, including using single rule exceptions, shared exception lists that apply to multiple rules, and value lists with exceptions.

Value lists and their values can be viewed and managed in Elastic Security. These are useful if you need to scale exceptions management or collect indicators of compromise to use in the indicator match rule.

Get an overview of how the detection rules are performing in your environment using the detection rule monitoring dashboard available in the Dashboards tab of Elastic Security.

This dashboard provides visualizations of rule execution statuses and time taken for rule execution. It also helps identify rule candidates for performance and query optimizations.

More information for monitoring and troubleshooting can be found in the Rule monitoring tab or in the Rule executions table of individual rules.

With 8.16, if you need to test rules over the past data or backfill missing alerts, you can do it with manual rule run. Manual run rule executions will happen with a lower priority when the system is not busy with scheduled rule runs. Once executed, the alerts will show up on the Alerts page, and if needed, you can filter for alerts from manual runs only.

Once the rules' performance monitoring is in place, detection teams should look to improve the quality of detection rules with rule reviews and automated testing and deployment. This is where Detection as Code concepts can be helpful.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.