IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Tutorial: Automate rollover with ILM Index lifecycle actions »
Elastic Docs ›Elasticsearch Guide [7.11] ›ILM: Manage the index lifecycle

Tutorial: Customize built-in ILM policies

edit

Tutorial: Customize built-in ILM policies

edit

Elasticsearch includes the following built-in ILM policies:

  • logs
  • metrics
  • synthetics

The Elastic Agent uses these policies to manage backing indices for its data streams. This tutorial shows you how to use Kibana’s Index Lifecycle Policies to customize these policies based on your application’s performance, resilience, and retention requirements.

Scenario

edit

You want to send log files to an Elasticsearch cluster so you can visualize and analyze the data. This data has the following retention requirements:

  • When the write index reaches 50GB or is 30 days old, roll over to a new index.
  • After rollover, keep indices in the hot data tier for 30 days.

  • 30 days after rollover:

    • Move indices to the warm data tier.
    • Set replica shards to 1.
    • Force merge multiple index segments to free up the space used by deleted documents.
  • Delete indices 90 days after rollover.

Prerequisites

edit

To complete this tutorial, you’ll need:

  • An Elasticsearch cluster with hot and warm data tiers.

    • Elasticsearch Service: Elastic Stack deployments on the Elasticsearch Service include a hot tier by default. To add a warm tier, edit your deployment and click Add capacity for the warm data tier.

      Add a warm data tier to your deployment

    • Self-managed cluster: Assign data_hot and data_warm roles to nodes as described in Data tiers.

      For example, include the data_warm node role in the elasticsearch.yml file of each node in the warm tier:

      node.roles: [ "data_warm" ]
  • A host with the Elastic Agent installed and configured to send logs to your Elasticsearch cluster.

View the policy

edit

The Elastic Agent uses data streams with an index pattern of logs-*-* to store log monitoring data. The built-in logs ILM policy automatically manages backing indices for these data streams.

To view the logs policy in Kibana:

  1. Open the menu and go to Stack Management > Index Lifecycle Policies.
  2. Select the logs policy.

The logs policy uses the recommended rollover defaults: Start writing to a new index when the current write index reaches 50GB or becomes 30 days old.

To view or change the rollover settings, click Advanced settings for the hot phase. Then disable Use recommended defaults to display the rollover settings.

View rollover defaults

Modify the policy

edit

The default logs policy is designed to prevent the creation of many tiny daily indices. You can modify the policy to meet your performance requirements and manage resource usage.

  1. Activate the warm phase and click Advanced settings.

    1. Disable Move to warm phase on rollover. Set Timing for warm phase to 30 days from rollover.
    2. Enable Set replicas and change Number of replicas to 1.
    3. Enable Force merge data and set Number of segments to 1.
    Add a warm phase with custom settings

  2. Activate the delete phase and set Timing for delete phase to 90 days from rollover.

    Add a delete phase
  3. Click Save Policy.
« Tutorial: Automate rollover with ILM Index lifecycle actions »