WARNING: Version 5.6 of the Elastic Stack has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Metric Functions Sum Functions »
Elastic Docs ›X-Pack for the Elastic Stack [5.6] ›Machine Learning in the Elastic Stack ›Function Reference

Rare Functions

edit

Rare Functions

edit

The rare functions detect values that occur rarely in time or rarely for a population.

The rare analysis detects anomalies according to the number of distinct rare values. This differs from freq_rare, which detects anomalies according to the number of times (frequency) rare values occur.

  • The rare and freq_rare functions should not be used in conjunction with exclude_frequent.
  • Shorter bucket spans (less than 1 hour, for example) are recommended when looking for rare events. The functions model whether something happens in a bucket at least once. With longer bucket spans, it is more likely that entities will be seen in a bucket and therefore they appear less rare. Picking the ideal the bucket span depends on the characteristics of the data with shorter bucket spans typically being measured in minutes, not hours.
  • To model rare data, a learning period of at least 20 buckets is required for typical data.

The X-Pack machine learning features include the following rare functions:

Rare

edit

The rare function detects values that occur rarely in time or rarely for a population. It detects anomalies according to the number of distinct rare values.

This function supports the following properties:

  • by_field_name (required)
  • over_field_name (optional)
  • partition_field_name (optional)

For more information about those properties, see Detector Configuration Objects.

Example 1: Analyzing status codes with the rare function.

{
  "function" : "rare",
  "by_field_name" : "status"
}

If you use this rare function in a detector in your job, it detects values that are rare in time. It models status codes that occur over time and detects when rare status codes occur compared to the past. For example, you can detect status codes in a web access log that have never (or rarely) occurred before.

Example 2: Analyzing status codes in a population with the rare function.

{
  "function" : "rare",
  "by_field_name" : "status",
  "over_field_name" : "clientip"
}

If you use this rare function in a detector in your job, it detects values that are rare in a population. It models status code and client IP interactions that occur. It defines a rare status code as one that occurs for few client IP values compared to the population. It detects client IP values that experience one or more distinct rare status codes compared to the population. For example in a web access log, a clientip that experiences the highest number of different rare status codes compared to the population is regarded as highly anomalous. This analysis is based on the number of different status code values, not the count of occurrences.

To define a status code as rare the X-Pack machine learning features look at the number of distinct status codes that occur, not the number of times the status code occurs. If a single client IP experiences a single unique status code, this is rare, even if it occurs for that client IP in every bucket.

Freq_rare

edit

The freq_rare function detects values that occur rarely for a population. It detects anomalies according to the number of times (frequency) that rare values occur.

This function supports the following properties:

  • by_field_name (required)
  • over_field_name (required)
  • partition_field_name (optional)

For more information about those properties, see Detector Configuration Objects.

Example 3: Analyzing URI values in a population with the freq_rare function.

{
  "function" : "freq_rare",
  "by_field_name" : "uri",
  "over_field_name" : "clientip"
}

If you use this freq_rare function in a detector in your job, it detects values that are frequently rare in a population. It models URI paths and client IP interactions that occur. It defines a rare URI path as one that is visited by few client IP values compared to the population. It detects the client IP values that experience many interactions with rare URI paths compared to the population. For example in a web access log, a client IP that visits one or more rare URI paths many times compared to the population is regarded as highly anomalous. This analysis is based on the count of interactions with rare URI paths, not the number of different URI path values.

To define a URI path as rare, the analytics consider the number of distinct values that occur and not the number of times the URI path occurs. If a single client IP visits a single unique URI path, this is rare, even if it occurs for that client IP in every bucket.

« Metric Functions Sum Functions »