Aggregating Data For Faster Performance

By default, datafeeds fetch data from Elasticsearch using search and scroll requests. It can be significantly more efficient, however, to aggregate data in Elasticsearch and to configure your jobs to analyze aggregated data.

One of the benefits of aggregating data this way is that Elasticsearch automatically distributes these calculations across your cluster. You can then feed this aggregated data into X-Pack machine learning instead of raw results, which reduces the volume of data that must be considered while detecting anomalies.

There are some limitations to using aggregations in datafeeds, however. Your aggregation must include a buckets aggregation, which in turn must contain a date histogram aggregation. This requirement ensures that the aggregated data is a time series. If you use a terms aggregation and the cardinality of a term is high, then the aggregation might not be effective and you might want to just use the default search and scroll behavior.

When you create or update a job, you can include the names of aggregations, for example:

PUT _xpack/ml/anomaly_detectors/farequote
{
  "analysis_config": {
    "bucket_span": "60m",
    "detectors": [{
      "function":"mean",
      "field_name":"responsetime",
      "by_field_name":"airline"
    }],
    "summary_count_field_name": "doc_count"
  },
  "data_description": {
    "time_field":"time"
  }
}

In this example, the airline, responsetime, and time fields are aggregations.

The aggregations are defined in the datafeed as follows:

PUT _xpack/ml/datafeeds/datafeed-farequote
{
  "job_id":"farequote",
  "indices": ["farequote"],
  "types": ["response"],
  "aggregations": {
    "buckets": {
      "date_histogram": {
        "field": "time",
        "interval": "360s",
        "time_zone": "UTC"
      },
      "aggregations": {
        "time": {
          "max": {"field": "time"}
        },
        "airline": {
          "terms": {
            "field": "airline",
            "size": 100
          },
          "aggregations": {
            "responsetime": {
              "avg": {
                "field": "responsetime"
              }
            }
          }
        }
      }
    }
  }
}

In this example, the aggregations have names that match the fields that they operate on. That is to say, the max aggregation is named time and its field is also time. The same is true for the aggregations with the names airline and responsetime. Since you must create the job before you can create the datafeed, synchronizing your aggregation and field names can simplify these configuration steps.

When you define an aggregation in a datafeed, it must have the following form:

"aggregations" : {
  "buckets" : {
    "date_histogram" : {
      "time_zone": "UTC", ...
    },
    "aggregations": {
      "<time_field>": {
        "max": {
          "field":"<time_field>"
        }
      }
      [,"<first_term>": {
        "terms":{...
        }
        [,"aggregations" : {
          [<sub_aggregation>]+
        } ]
      }]
   }
 }
}

You must specify buckets as the aggregation name and date_histogram as the aggregation type. For more information, see Date Histogram Aggregation.

Each histogram bucket has a key, which is the bucket start time. This key cannot be used for aggregations in datafeeds, however, because they need to know the time of the latest record within a bucket. Otherwise, when you restart a datafeed, it continues from the start time of the histogram bucket and possibly fetches the same data twice. The max aggregation for the time field is therefore necessary to provide the time of the latest record within a bucket.

You can optionally specify a terms aggregation, which creates buckets for different values of a field.

You can change this behavior by setting the size parameter. To determine the cardinality of your data, you can run searches such as:

GET .../_search {
  "aggs": {
    "service_cardinality": {
      "cardinality": {
        "field": "service"
        }
    }
  }
}

By default, Elasticsearch limits the maximum number of terms returned to 10000. For high cardinality fields, the query might not run. It might return errors related to circuit breaking exceptions that indicate that the data is too large. In such cases, do not use aggregations in your datafeed. For more information, see Terms Aggregation.

You can also optionally specify multiple sub-aggregations. The sub-aggregations are aggregated for the buckets that were created by their parent aggregation. For more information, see Aggregations.