Dynamic Index Names

Several watch constructs deal with indices, including index action, the search transform and the search input. When configuring these constructs you can set the index names to static values. In addition to specifying static index names, Watcher enables you to specify indexes using dynamic time-aware templates. These templates resolve to specific index names during the watch execution according to the execution time.

Dynamic index name resolution enables you to search a range of time-series indices, rather than searching all of your time-series indices and filtering the the results. Limiting the number of indices that are searched reduces the load on the cluster and improves watch execution performance. For example, if you are using a watch to monitor errors in your daily logs, you can use a dynamic index name template to restrict the search to the past two days.

A dynamic index name takes the following form:

<static_name{date_math_expr{date_format}}>

Where:

The following example shows different forms of dynamic index names and the final index names they resolve to given the execution date is 22rd March 2024.

To use the characters { and } in the static part of an index name template, escape them with a backslash, \:

The following example shows a search input that searches the Logstash indices for the past three days, assuming the indices use the default Logstash index name format, logstash-YYYY.MM.dd.

{
  ...
  "input" : {
    "search" : {
      "request" : {
        "indices" : [
          "<logstash-{now/d-2d}>",
          "<logstash-{now/d-1d}>",
          "<logstash-{now/d}>"
        ],
        ...
      }
    }
  }
  ...
}

By default, the index names are resolved base on UTC time zone. You can change this default at multiple levels:

Configuring the following setting set the default dynamic index name time zone in watcher:

watcher.dynamic_indices.time_zone: '+01:00'

You can also configure the default time zone separately on each of the construct that make use of it (search input/transform and index action):

watcher.input.search.dynamic_indices.time_zone: '+01:00'

watcher.transform.search.dynamic_indices.time_zone: '+01:00'

watcher.actions.index.dynamic_indices.time_zone: '+01:00'

Alternatively, each of these construct can define their own time zone within the watch definition.