IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Users page Prebuilt job reference »
Elastic Docs ›Elastic Security Solution [8.8]

Anomaly detection with machine learning

edit

Anomaly detection with machine learning

edit

Machine learning functionality is available when you have the appropriate subscription, are using a cloud deployment, or are testing out a Free Trial. Refer to Machine learning job and rule requirements.

You can view the details of detected anomalies within the Anomalies table widget shown on the Hosts, Network, and associated details pages, or even narrow to the specific date range of an anomaly from the Max anomaly score by job field in the overview of the details pages for hosts and IPs. These interfaces also offer the ability to drag and drop details of the anomaly to Timeline, such as the Entity itself, or any of the associated Influencers.

Manage machine learning jobs

edit

If you have the machine_learning_admin role, you can use the ML job settings interface on the Alerts, Rules, and Rule Exceptions pages to view, start, and stop Elastic Security machine learning jobs.

ML job settings UI on the Alerts page

To add a custom job to the ML job settings interface, add Security to the job’s Groups field (Kibana → Machine learning → Create/Edit job → Job details).

Manage machine learning detection rules

edit

You can also check the status of machine learning detection rules, and start or stop their associated machine learning jobs:

  • In the Rules table, the Last response column displays the rule’s current status. An indicator icon (Error icon from Rules table) also appears if a required machine learning job isn’t running. Click the icon to list the affected jobs, then click Visit rule details page to investigate to open the rule’s details page.

    Rules table machine learning job error

  • On a rule’s details page, check the Definition section to confirm whether the required machine learning jobs are running. Switch the toggles on or off to run or stop each job.

    Rule details page with ML job stopped

Prebuilt jobs

edit

Elastic Security comes with prebuilt machine learning anomaly detection jobs for automatically detecting host and network anomalies. The jobs are displayed in the Anomaly Detection interface. They are available when either:

  • You ship data using Beats or the Elastic Agent, and Kibana is configured with the required index patterns (such as auditbeat-*, filebeat-*, packetbeat-*, or winlogbeat-* in Kibana → Stack Management → Data Views).

Or

  • Your shipped data is ECS-compliant, and Kibana is configured with the shipped data’s index patterns in Kibana → Stack Management → Data Views.

Prebuilt job reference describes all available machine learning jobs and lists which ECS fields are required on your hosts when you are not using Beats or the Elastic Agent to ship your data. For information on tuning anomaly results to reduce the number of false positives, see Optimizing anomaly results.

Machine learning jobs look back and analyze two weeks of historical data prior to the time they are enabled. After jobs are enabled, they continuously analyze incoming data. When jobs are stopped and restarted within the two-week time frame, previously analyzed data is not processed again.

View detected anomalies

edit

To view the Anomalies table widget and Max Anomaly Score By Job details, the user must have the machine_learning_admin or machine_learning_user role.

To adjust the score threshold that determines which anomalies are shown, you can modify Kibana → Stack Management → Advanced Settings → securitySolution:defaultAnomalyScore.

« Users page Prebuilt job reference »