About detection rules
editAbout detection rules
editRules run periodically and search for source events, matches, sequences, or machine learning job anomaly results that meet their criteria. When a rule’s criteria are met, a detection alert is created.
Rule types
editYou can create the following types of rules:
- Custom query: Query-based rule, which searches the defined indices and creates an alert when one or more documents match the rule’s query.
-
Machine learning: Machine learning rule, which creates an alert when a machine learning job discovers an anomaly above the defined threshold (see Anomaly detection).
For machine learning rules, the associated machine learning job must be running. If the machine learning job isn’t running, the rule will:
- Run and create alerts if existing anomaly results with scores above the defined threshold are discovered.
- Issue an error stating the machine learning job was not running when the rule executed.
-
Threshold: Searches the defined indices and creates a detections alert when the number of times the specified field’s value is present and meets the threshold during a single execution. When multiple values meet the threshold, an alert is generated for each value.
For example, if the threshold
fieldissource.ipand itsvalueis10, an alert is generated for every source IP address that appears in at least 10 of the rule’s search results. - Event correlation: Searches the defined indices and creates an alert when results match an Event Query Language (EQL) query.
-
Indicator match: Creates an alert when Elastic Security index field values match field values defined in the specified indicator index patterns. For example, you can create an indicator index for IP addresses and use this index to create an alert whenever an event’s
destination.ipequals a value in the index. Indicator index field mappings should be ECS-compliant. For information on creating Elasticsearch indices and field types, see Index some documents, Create index API, and Field data types. If you have indicators in a standard file format, such as CSV or JSON, you can also use the Machine Learning Data Visualizer to import your indicators into an indicator index. See Explore the data in Kibana and use the Import Data option to import your indicators.You can also use value lists as the indicator match index. See Use value lists with indicator match rules at the end of this topic for more information.
-
New terms: Generates an alert for each new term detected in source documents within a specified time range. You can also detect a combination of up to three new terms (for example, a
host.ipandhost.idthat have never been observed together before). -
ES|QL: Searches the defined indices and creates an alert when results match an Elasticsearch Query Language (ES|QL) query.
ES|QL is enabled by default in Kibana. It can be disabled using the
enableESQLsetting from the Advanced Settings. This will hide the ES|QL user interface from various applications. However, users will be able to access existing ES|QL artifacts like saved searches and visualizations.
Data views and index patterns
editWhen you create a rule, you must either specify the Elasticsearch index pattens for which you’d like the rule to run, or select a data view field as the data source. If you select a data view, you can select runtime fields associated with that data view to create a query for the rule (with the exception of machine learning rules, which do not use queries).
To access data views, ensure you have the required permissions.
Notifications
editFor both prebuilt and custom rules, you can send notifications when alerts are created. Notifications can be sent via Jira, Microsoft Teams, PagerDuty, Slack, and others, and can be configured when you create or edit a rule.
Authorization
editRules, including all background detection and the actions they generate, are authorized using an API key associated with the last user to edit the rule. Upon creating or modifying a rule, an API key is generated for that user, capturing a snapshot of their privileges. The API key is then used to run all background tasks associated with the rule including detection checks and executing actions.
If a rule requires certain privileges to run, such as index privileges, keep in mind that if a user without those privileges updates the rule, the rule will no longer function.
Exceptions
editWhen modifying rules or managing detection alerts, you can add exceptions that prevent a rule from generating alerts even when its criteria are met. This is useful for reducing noise, such as preventing alerts from trusted processes and internal IP addresses.
You can add exceptions to custom query, machine learning, event correlation, and indicator match rule types.