Rollup search limitations

While we feel the Rollup function is extremely flexible, the nature of summarizing data means there will be some limitations. Once live data is thrown away, you will always lose some flexibility.

This page highlights the major limitations so that you are aware of them.

When using the Rollup search endpoint, the index parameter accepts one or more indices. These can be a mix of regular, non-rollup indices and rollup indices. However, only one rollup index can be specified. The exact list of rules for the index parameter are as follows:

This limitation is driven by the logic that decides which jobs are the "best" for any given query. If you have ten jobs stored in a single index, which cover the source data with varying degrees of completeness and different intervals, the query needs to determine which set of jobs to actually search. Incorrect decisions can lead to inaccurate aggregation results (e.g. over-counting doc counts, or bad metrics). Needless to say, this is a technically challenging piece of code.

To help simplify the problem, we have limited search to just one rollup index at a time (which may contain multiple jobs). In the future we may be able to open this up to multiple rollup jobs.

A perhaps obvious limitation, but rollups can only aggregate on data that has been stored in the rollups. If you don’t configure the rollup job to store metrics about the price field, you won’t be able to use the price field in any query or aggregation.

For example, the temperature field in the following query has been stored in a rollup job…​ but not with an avg metric. Which means the usage of avg here is not allowed:

GET sensor_rollup/_rollup_search
{
  "size": 0,
  "aggregations": {
    "avg_temperature": {
      "avg": {
        "field": "temperature"
      }
    }
  }
}

The response will tell you that the field and aggregation were not possible, because no rollup jobs were found which contained them:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "There is not a rollup job that has a [avg] agg with name [avg_temperature] which also satisfies all requirements of query.",
        "stack_trace": ...
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "There is not a rollup job that has a [avg] agg with name [avg_temperature] which also satisfies all requirements of query.",
    "stack_trace": ...
  },
  "status": 400
}

Rollups are stored at a certain granularity, as defined by the date_histogram group in the configuration. This means you can only search/aggregate the rollup data with an interval that is greater-than or equal to the configured rollup interval.

For example, if data is rolled up at hourly intervals, the Rollup search API can aggregate on any time interval hourly or greater. Intervals that are less than an hour will throw an exception, since the data simply doesn’t exist for finer granularities.

Because the RollupSearch endpoint can "upsample" intervals, there is no need to configure jobs with multiple intervals (hourly, daily, etc). It’s recommended to just configure a single job with the smallest granularity that is needed, and allow the search endpoint to upsample as needed.

That said, if multiple jobs are present in a single rollup index with varying intervals, the search endpoint will identify and use the job(s) with the largest interval to satisfy the search request.

The Rollup functionality allows query's in the search request, but with a limited subset of components. The queries currently allowed are:

Furthermore, these queries can only use fields that were also saved in the rollup job as a group. If you wish to filter on a keyword hostname field, that field must have been configured in the rollup job under a terms grouping.

If you attempt to use an unsupported query, or the query references a field that wasn’t configured in the rollup job, an exception will be thrown. We expect the list of support queries to grow over time as more are implemented.

Rollup documents are stored in the timezone of the date_histogram group configuration in the job. If no timezone is specified, the default is to rollup timestamps in UTC.