IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Getting started with rollups Rollup aggregation limitations »

› › ›

Understanding groups

edit

Understanding groups

edit

This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

To preserve flexibility, Rollup Jobs are defined based on how future queries may need to use the data. Traditionally, systems force the admin to make decisions about what metrics to rollup and on what interval. E.g. The average of cpu_time on an hourly basis. This is limiting; if, at a future date, the admin wishes to see the average of cpu_time on an hourly basis and partitioned by `host_name`, they are out of luck.

Of course, the admin can decide to rollup the [hour, host] tuple on an hourly basis, but as the number of grouping keys grows, so do the number of tuples the admin needs to configure. Furthermore, these [hours, host] tuples are only useful for hourly rollups… daily, weekly, or monthly rollups all require new configurations.

Rather than force the admin to decide ahead of time which individual tuples should be rolled up, Elasticsearch’s Rollup jobs are configured based on which groups are potentially useful to future queries. For example, this configuration:

"groups" : {
  "date_histogram": {
    "field": "timestamp",
    "fixed_interval": "1h",
    "delay": "7d"
  },
  "terms": {
    "fields": ["hostname", "datacenter"]
  },
  "histogram": {
    "fields": ["load", "net_in", "net_out"],
    "interval": 5
  }
}

Allows date_histogram's to be used on the "timestamp" field, terms aggregations to be used on the "hostname" and "datacenter" fields, and histograms to be used on any of "load", "net_in", "net_out" fields.

Importantly, these aggs/fields can be used in any combination. This aggregation:

"aggs" : {
  "hourly": {
    "date_histogram": {
      "field": "timestamp",
      "fixed_interval": "1h"
    },
    "aggs": {
      "host_names": {
        "terms": {
          "field": "hostname"
        }
      }
    }
  }
}

is just as valid as this aggregation:

"aggs" : {
  "hourly": {
    "date_histogram": {
      "field": "timestamp",
      "fixed_interval": "1h"
    },
    "aggs": {
      "data_center": {
        "terms": {
          "field": "datacenter"
        }
      },
      "aggs": {
        "host_names": {
          "terms": {
            "field": "hostname"
          }
        },
        "aggs": {
          "load_values": {
            "histogram": {
              "field": "load",
              "interval": 5
            }
          }
        }
      }
    }
  }
}

You’ll notice that the second aggregation is not only substantially larger, it also swapped the position of the terms aggregation on "hostname", illustrating how the order of aggregations does not matter to rollups. Similarly, while the date_histogram is required for rolling up data, it isn’t required while querying (although often used). For example, this is a valid aggregation for Rollup Search to execute:

"aggs" : {
  "host_names": {
    "terms": {
      "field": "hostname"
    }
  }
}

Ultimately, when configuring groups for a job, think in terms of how you might wish to partition data in a query at a future date… then include those in the config. Because Rollup Search allows any order or combination of the grouped fields, you just need to decide if a field is useful for aggregating later, and how you might wish to use it (terms, histogram, etc).

Calendar vs fixed time intervals

edit

Each rollup-job must have a date histogram group with a defined interval. Elasticsearch understands both calendar and fixed time intervals. Fixed time intervals are fairly easy to understand; 60s means sixty seconds. But what does 1M mean? One month of time depends on which month we are talking about, some months are longer or shorter than others. This is an example of calendar time and the duration of that unit depends on context. Calendar units are also affected by leap-seconds, leap-years, etc.

This is important because the buckets generated by rollup are in either calendar or fixed intervals and this limits how you can query them later. See Requests must be multiples of the config.

We recommend sticking with fixed time intervals, since they are easier to understand and are more flexible at query time. It will introduce some drift in your data during leap-events and you will have to think about months in a fixed quantity (30 days) instead of the actual calendar length. However, it is often easier than dealing with calendar units at query time.

Multiples of units are always "fixed". For example, 2h is always the fixed quantity 7200 seconds. Single units can be fixed or calendar depending on the unit:

Unit	Calendar	Fixed
millisecond	NA	`1ms`, `10ms`, etc
second	NA	`1s`, `10s`, etc
minute	`1m`	`2m`, `10m`, etc
hour	`1h`	`2h`, `10h`, etc
day	`1d`	`2d`, `10d`, etc
week	`1w`	NA
month	`1M`	NA
quarter	`1q`	NA
year	`1y`	NA

For some units where there are both fixed and calendar, you may need to express the quantity in terms of the next smaller unit. For example, if you want a fixed day (not a calendar day), you should specify 24h instead of 1d. Similarly, if you want fixed hours, specify 60m instead of 1h. This is because the single quantity entails calendar time, and limits you to querying by calendar time in the future.

Grouping limitations with heterogeneous indices

edit

There was previously a limitation in how Rollup could handle indices that had heterogeneous mappings (multiple, unrelated/non-overlapping mappings). The recommendation at the time was to configure a separate job per data "type". For example, you might configure a separate job for each Beats module that you had enabled (one for process, another for filesystem, etc).

This recommendation was driven by internal implementation details that caused document counts to be potentially incorrect if a single "merged" job was used.

This limitation has since been alleviated. As of 6.4.0, it is now considered best practice to combine all rollup configurations into a single job.

As an example, if your index has two types of documents:

{
  "timestamp": 1516729294000,
  "temperature": 200,
  "voltage": 5.2,
  "node": "a"
}

and

{
  "timestamp": 1516729294000,
  "price": 123,
  "title": "Foo"
}

the best practice is to combine them into a single rollup job which covers both of these document types, like this:

PUT _rollup/job/combined
{
  "index_pattern": "data-*",
  "rollup_index": "data_rollup",
  "cron": "*/30 * * * * ?",
  "page_size": 1000,
  "groups": {
    "date_histogram": {
      "field": "timestamp",
      "fixed_interval": "1h",
      "delay": "7d"
    },
    "terms": {
      "fields": [ "node", "title" ]
    }
  },
  "metrics": [
    {
      "field": "temperature",
      "metrics": [ "min", "max", "sum" ]
    },
    {
      "field": "price",
      "metrics": [ "avg" ]
    }
  ]
}

Doc counts and overlapping jobs

edit

There was previously an issue with document counts on "overlapping" job configurations, driven by the same internal implementation detail. If there were two Rollup jobs saving to the same index, where one job is a "subset" of another job, it was possible that document counts could be incorrect for certain aggregation arrangements.

This issue has also since been eliminated in 6.4.0.

« Getting started with rollups Rollup aggregation limitations »