Understanding groups

Each rollup-job must have a date histogram group with a defined interval. Elasticsearch understands both calendar and fixed time intervals. Fixed time intervals are fairly easy to understand; 60s means sixty seconds. But what does 1M mean? One month of time depends on which month we are talking about, some months are longer or shorter than others. This is an example of calendar time and the duration of that unit depends on context. Calendar units are also affected by leap-seconds, leap-years, etc.

This is important because the buckets generated by rollup are in either calendar or fixed intervals and this limits how you can query them later. See Requests must be multiples of the config.

We recommend sticking with fixed time intervals, since they are easier to understand and are more flexible at query time. It will introduce some drift in your data during leap-events and you will have to think about months in a fixed quantity (30 days) instead of the actual calendar length. However, it is often easier than dealing with calendar units at query time.

Multiples of units are always "fixed". For example, 2h is always the fixed quantity 7200 seconds. Single units can be fixed or calendar depending on the unit:

For some units where there are both fixed and calendar, you may need to express the quantity in terms of the next smaller unit. For example, if you want a fixed day (not a calendar day), you should specify 24h instead of 1d. Similarly, if you want fixed hours, specify 60m instead of 1h. This is because the single quantity entails calendar time, and limits you to querying by calendar time in the future.

There was previously a limitation in how Rollup could handle indices that had heterogeneous mappings (multiple, unrelated/non-overlapping mappings). The recommendation at the time was to configure a separate job per data "type". For example, you might configure a separate job for each Beats module that you had enabled (one for process, another for filesystem, etc).

This recommendation was driven by internal implementation details that caused document counts to be potentially incorrect if a single "merged" job was used.

This limitation has since been alleviated. As of 6.4.0, it is now considered best practice to combine all rollup configurations into a single job.

As an example, if your index has two types of documents:

{
  "timestamp": 1516729294000,
  "temperature": 200,
  "voltage": 5.2,
  "node": "a"
}

and

{
  "timestamp": 1516729294000,
  "price": 123,
  "title": "Foo"
}

the best practice is to combine them into a single rollup job which covers both of these document types, like this:

PUT _rollup/job/combined
{
    "index_pattern": "data-*",
    "rollup_index": "data_rollup",
    "cron": "*/30 * * * * ?",
    "page_size" :1000,
    "groups" : {
      "date_histogram": {
        "field": "timestamp",
        "fixed_interval": "1h",
        "delay": "7d"
      },
      "terms": {
        "fields": ["node", "title"]
      }
    },
    "metrics": [
        {
            "field": "temperature",
            "metrics": ["min", "max", "sum"]
        },
        {
            "field": "price",
            "metrics": ["avg"]
        }
    ]
}

There was previously an issue with document counts on "overlapping" job configurations, driven by the same internal implementation detail. If there were two Rollup jobs saving to the same index, where one job is a "subset" of another job, it was possible that document counts could be incorrect for certain aggregation arrangements.

This issue has also since been eliminated in 6.4.0.