New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
A newer version is available. For the latest information, see the current release documentation.
« reverse state »
Elastic Docs ›Curator Reference [5.5] ›Filter Elements

source

edit
IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.

source

edit

The source from which to derive the index or snapshot age. Can be one of name, creation_date, or field_stats.

This setting is only used with the age filtertype, or
with the space filtertype when use_age is set to True.

When using the age filtertype, source requires
direction, unit, unit_count,
and additionally, the optional setting, epoch.

name-based ages

edit

Using name as the source tells Curator to look for a timestring within the index or snapshot name, and convert that into an epoch timestamp (epoch implies UTC).

 - filtertype: age
   source: name
   direction: older
   timestring: '%Y.%m.%d'
   unit: days
   unit_count: 3

A word about regular expression matching with timestrings

Timestrings are parsed from strftime patterns, like %Y.%m.%d, into regular expressions. For example, %Y is 4 digits, so the regular expression for that looks like \d{4}, and %m is 2 digits, so the regular expression is \d{2}.

What this means is that a simple timestring to match year and month, %Y.%m will result in a regular expression like this: ^.*\d{4}\.\d{2}.*$. This pattern will match any 4 digits, followed by a period ., followed by 2 digits, occurring anywhere in the index name. This means it will match monthly indices, like index-2016.12, as well as daily indices, like index-2017.04.01, which may not be the intended behavior.

To compensate for this, when selecting indices matching a subset of another pattern, use a second filter with exclude set to True

- filtertype: pattern
 kind: timestring
 value: '%Y.%m'
- filtertype: pattern
 kind: timestring
 value: '%Y.%m.%d'
 exclude: True

This will prevent the %Y.%m pattern from matching the %Y.%m part of the daily indices.

This applies whether using timestring as a mere pattern match, or as part of date calculations.

creation_date-based ages

edit

creation_date extracts the epoch time of index or snapshot creation.

 - filtertype: age
   source: creation_date
   direction: older
   unit: days
   unit_count: 3

field_stats-based ages

edit

source can only be field_stats when filtering indices.

In Curator 5.3 and older, source field_stats uses the Field Stats API to calculate either the min_value or the max_value of the field as the stats_result, and then use that value for age comparisons. In 5.4 and above, even though it is still called field_stats, it uses an aggregation to calculate the same values, as the field_stats API is no longer used in Elasticsearch 6.x and up.

field must be of type date in Elasticsearch.

 - filtertype: age
   source: field_stats
   direction: older
   unit: days
   unit_count: 3
   field: '@timestamp'
   stats_result: min_value
« reverse state »

On this page

Was this helpful?
Feedback