Machine learning anomaly detection limitations

The following limitations and known problems apply to the 7.10.2 release of the Elastic machine learning features. The limitations are grouped into four categories:

Machine learning uses Streaming SIMD Extensions (SSE) 4.2 instructions, so it works only on machines whose CPUs support SSE4.2. If you run Elasticsearch on older hardware you must disable machine learning by setting xpack.ml.enabled to false. See Machine learning settings in Elasticsearch.

When there are many machine learning jobs running at the same time and there are insufficient CPU resources, the JVM performance must be prioritized so search and indexing latency remain acceptable. To that end, when CPU is constrained on Linux and MacOS environments, the CPU scheduling priority of native analysis processes is reduced to favor the Elasticsearch JVM. This improvement does not apply to Windows environments.

By default, the terms aggregation returns the buckets for the top ten terms. You can change this default behavior by setting the size parameter.

If you send pre-aggregated data to a job for analysis, you must ensure that the size is configured correctly. Otherwise, some data might not be analyzed.

You cannot use the following field names in the by_field_name or over_field_name properties in a job: by; count; over. This limitation also applies to those properties when you create advanced jobs in Kibana.

Rollup indices and index patterns cannot be used in machine learning jobs or datafeeds. This limitation applies irrespective of whether you create the jobs in Kibana or by using APIs. In Kibana, if you select an index, saved search, or index pattern that uses the Rollup feature, the machine learning job creation wizards fail.

See Rolling up historical data.

Frozen indices cannot be used in anomaly detection jobs or datafeeds. This limitation applies irrespective of whether you create the jobs in Kibana or by using APIs. This limitation exists because it’s currently not possible to specify the ignore_throttled query parameter for search requests in datafeeds or jobs. See Searching a frozen index.

There are some limitations that affect your ability to create a forecast:

Categorization identifies static parts of unstructured logs and groups similar messages together. The default categorization tokenizer assumes English language log messages. For other languages you must define a different categorization_analyzer for your job. For more information, see Detecting anomalous categories of data.

Additionally, a dictionary used to influence the categorization process contains only English words. This means categorization might work better in English than in other languages. The ability to customize the dictionary will be added in a future release.

The post data API enables you to send data to a job for analysis. The data that you send to the job must use the JSON format.

For more information about this API, see Post Data to Jobs.

One of the counts associated with a machine learning job is missing_field_count, which indicates the number of records that are missing a configured field.

Since jobs analyze JSON data, the missing_field_count might be misleading. Missing fields might be expected due to the structure of the data and therefore do not generate poor results.

For more information about missing_field_count, see the get anomaly detection job statistics API.

When the Elasticsearch security features are enabled, a datafeed stores the roles of the user who created or updated the datafeed at that time. This means that if the roles the user has are changed after they create or update a datafeed then the datafeed continues to run without change. However, if instead the permissions associated with the roles that are stored with the datafeed are changed then this affects the datafeed. For more information, see Datafeeds.

In 6.6 and later releases, the get jobs API and the get job statistics API return a maximum of 10,000 jobs. Likewise, the get datafeeds API and the get datafeed statistics API return a maximum of 10,000 datafeeds.

There are some factors that may be considered when you run forecasts:

The machine learning features in Kibana use pop-ups. You must configure your web browser so that it does not block pop-up windows or create an exception for your Kibana URL.

In Kibana, Anomaly Explorer and Single Metric Viewer charts are not displayed for anomalies that were due to categorization (if model plot is not enabled), time_of_day functions, time_of_week functions, or lat_long geographic functions.

If model plot is not enabled, the charts are not displayed for detectors that use script fields either (except for scripts that define metric fields). In that case, the original source data cannot be easily searched because it has been transformed by the script.

If your datafeed uses aggregations with nested terms aggs and model plot is not enabled for the anomaly detection job, neither the Anomaly Explorer nor the Single Metric Viewer can plot and display an anomaly chart for the job. In these cases, the charts are not visible and an explanatory message is shown.

The charts can also look odd in circumstances where there is very little data to plot. For example, if there is only one data point, it is represented as a single dot. If there are only two data points, they are joined by a line.

If you create jobs in Kibana, you must use datafeeds. If the data that you want to analyze is not stored in Elasticsearch, you cannot use datafeeds and therefore you cannot create your jobs in Kibana. You can, however, use the machine learning APIs to create jobs and to send batches of data directly to the jobs. For more information, see Datafeeds and API quick reference.

If you create single or multi-metric jobs in Kibana, it might enable some options under the covers that you’d want to reconsider for large or long-running jobs.

For example, when you create a single metric job in Kibana, it generally enables the model_plot_config advanced configuration option. That configuration option causes model information to be stored along with the results and provides a more detailed view into anomaly detection. It is specifically used by the Single Metric Viewer in Kibana. When this option is enabled, however, it can add considerable overhead to the performance of the system. If you have jobs with many entities, for example data from tens of thousands of servers, storing this additional model information for every bucket might be problematic. If you are not certain that you need this option or if you experience performance issues, edit your job configuration to disable this option.

Likewise, when you create a single or multi-metric job in Kibana, in some cases it uses aggregations on the data that it retrieves from Elasticsearch. One of the benefits of summarizing data this way is that Elasticsearch automatically distributes these calculations across your cluster. This summarized data is then fed into machine learning instead of raw results, which reduces the volume of data that must be considered while detecting anomalies. However, if you have two jobs, one of which uses pre-aggregated data and another that does not, their results might differ. This difference is due to the difference in precision of the input data. The machine learning analytics are designed to be aggregation-aware and the likely increase in performance that is gained by pre-aggregating the data makes the potentially poorer precision worthwhile. If you want to view or change the aggregations that are used in your job, refer to the aggregations property in your datafeed.

When the aggregation interval of the datafeed and the bucket span of the job don’t match, the values of the chart plotted in both the Single Metric Viewer and the Anomaly Explorer differ from the actual values of the job. To avoid this behavior, make sure that the aggregation interval in the datafeed configuration and the bucket span in the anomaly detection job configuration have the same values.

If you create spaces in Kibana, you see only the saved objects that belong to that space. This limited scope does not apply to machine learning objects; they are visible in all of your spaces.

However, the Kibana machine learning features interact with some saved objects (such as index patterns, dashboards, and visualizations) that might not be available in all spaces. For example: