IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Manage existing indices Reindexing via ILM »
Elastic Docs ›Elasticsearch Guide [7.6] ›ILM: Manage the index lifecycle ›Manage existing indices

Managing existing periodic indices with ILM

edit

Managing existing periodic indices with ILM

edit

The examples in this section assume daily indices as set up in the previous section.

The simplest way to manage existing indices while transitioning to fully ILM-managed indices is to allow all new indices to be fully managed by ILM before attaching ILM policies to existing indices. To do this, all new documents should be directed to ILM-managed indices - if you are using Beats or Logstash data shippers, upgrading all of those shippers to version 7.0.0 or higher will take care of that part for you. If you are not using Beats or Logstash, you may need to set up ILM for new indices yourself as demonstrated in the ILM tutorial.

If you are using Beats through Logstash, you may need to change your Logstash output configuration and invoke the Beats setup to use ILM for new data.

Once all new documents are being written to fully ILM-managed indices, it is easy to add an ILM policy to existing indices. However, there are two things to keep in mind when doing this, and a trick that makes those two things much easier to handle.

The two biggest things to keep in mind are:

  1. Existing periodic indices shouldn’t use policies with rollover, because rollover is used to manage where new data goes. Since existing indices should no longer be receiving new documents, there is no point to using rollover for them.
  2. ILM policies attached to existing indices will compare the min_age for each phase to the original creation date of the index, and so might proceed through multiple phases immediately.

The first one is the most important, because it makes it difficult to use the same policy for new and existing periodic indices. But that’s easy to solve with one simple trick: Create a second policy for existing indices, in addition to the one for new indices. ILM policies are cheap to create, so don’t be afraid to have more than one. Modifying a policy designed for new indices to be used on existing indices is generally very simple: just remove the rollover action.

For example, if you created a policy for your new indices with each phase like so:

PUT _ilm/policy/mylogs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "25GB"
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "7d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

You can create a policy for pre-existing indices by removing the rollover action, and in this case, the hot phase is now empty so we can remove that too:

PUT _ilm/policy/mylogs_policy_existing
{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "1d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "7d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Creating a separate policy for existing indices will also allow using different min_age values. You may want to use higher values to prevent many indices from running through the policy at once, which may be important if your policy includes potentially resource-intensive operations like force merge.

You can configure the lifecycle for many indices at once by using wildcards in the index name when calling the Update Settings API to set the policy name, but be careful that you don’t include any indices that you don’t want to change the policy for:

PUT mylogs-pre-ilm*/_settings 
{
  "index": {
    "lifecycle": {
      "name": "mylogs_policy_existing"
    }
  }
}

This pattern will match all indices with names that start with mylogs-pre-ilm

Once all pre-ILM indices have aged out and been deleted, the policy for older periodic indices can be deleted.

« Manage existing indices Reindexing via ILM »