New features in Curator 4.1

Configuration Validation

Curator 4.0 did not validate configuration options.  As a result, some users had a bad experience — like having all of their indices selected, instead of a subset.  So the first new feature in Curator 4.1 aims to prevent those sorts of situations with Configuration Validation.  This was a bigger project than I’d originally anticipated, but I wanted to get it right.  As a result, Curator will now fail to run if any option is configured outside of acceptable parameters.  This does not prevent something from running if you set a 2 instead of a 3, for example, but other typos will be caught immediately.  As always, it is recommended that you first test any new or changed configuration with the --dry-run flag for safety.

Filter by Alias

Curator 4.1 now allows you to select indices based on their inclusion in an alias. Or even multiple aliases!

- filtertype: alias  
  aliases: [ alias_one, alias_two ]
  exclude: False

Filter by Count

You can now also filter by count, or in other words, keep the most recent n indices or snapshots (or the earliest, you choose).  It uses similar configuration to the space filter, in that you can specify to sort by alphanumeric (the default) or by age.

- filtertype: count
  count: 10
  use_age: True
  source: creation_date

This filter configuration will preserve the 10 most recent items by removing them from the actionable list.  You would likely use this in conjunction with the pattern filter to keep only the n most recent items matching a given pattern, like this:

filters:
- filtertype: pattern
  kind: prefix
  value: 'logstash-'
- filtertype: count
  count: 10
  use_age: True
  source: creation_date

These filters will keep the 10 most recent indices starting with logstash-, based on the index creation_date, and perform the selected action on all other indices starting with logstash-.

Environment Variables

Also new in Curator 4.1 is the ability to use environment variables in both your configuration and action files.  This feature is experimental, so please use caution when testing it out.  When in doubt, always use the --dry-run flag.

To use an environment variable in a configuration file, use:

${VAR}

Where VAR is the name of the environment variable.

Each variable reference is replaced at startup by the value of the environment variable. The replacement is case-sensitive and occurs while the YAML file is parsed, but before configuration schema validation. References to undefined variables are replaced by None unless you specify a default value. To specify a default value, use:

${VAR:default_value}

Where default_value is the value to use if the environment variable is undefined.

Please note that environment variables are only available for values in YAML and not for keys. In other words, you can do this:

key: ${ENVVAR}

But not this:

${ENVVAR}: value

Conclusion

These new features open the door to even more possibilities with Curator. I can’t wait to hear how you’re using them! As always, if you encounter an issue, you are always welcome to find help in our discussion forum, or submit any issue you may encounter at our GitHub repository.

Happy Curating!