Managing existing periodic indices with ILM

The simplest way to manage existing indices while transitioning to fully ILM-managed indices is to allow all new indices to be fully managed by ILM before attaching ILM policies to existing indices. To do this, all new documents should be directed to ILM-managed indices - if you are using Beats or Logstash data shippers, upgrading all of those shippers to version 7.0.0 or higher will take care of that part for you. If you are not using Beats or Logstash, you may need to set up ILM for new indices yourself as demonstrated in the getting started guide.

Once all new documents are being written to fully ILM-managed indices, it is easy to add an ILM policy to existing indices. However, there are two things to keep in mind when doing this, and a trick that makes those two things much easier to handle.

The two biggest things to keep in mind are:

The first one is the most important, because it makes it difficult to use the same policy for new and existing periodic indices. But that’s easy to solve with one simple trick: Create a second policy for existing indices, in addition to the one for new indices. ILM policies are cheap to create, so don’t be afraid to have more than one. Modifying a policy designed for new indices to be used on existing indices is generally very simple: just remove the rollover action.

For example, if you created a policy for your new indices with each phase like so:

PUT _ilm/policy/mylogs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "25GB"
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "7d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

You can create a policy for pre-existing indices by removing the rollover action, and in this case, the hot phase is now empty so we can remove that too:

PUT _ilm/policy/mylogs_policy_existing
{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "1d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "7d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Creating a separate policy for existing indices will also allow using different min_age values. You may want to use higher values to prevent many indices from running through the policy at once, which may be important if your policy includes potentially resource-intensive operations like force merge.

You can configure the lifecycle for many indices at once by using wildcards in the index name when calling the Update Settings API to set the policy name, but be careful that you don’t include any indices that you don’t want to change the policy for:

PUT mylogs-pre-ilm*/_settings 
{
  "index": {
    "lifecycle": {
      "name": "mylogs_policy_existing"
    }
  }
}

Once all pre-ILM indices have aged out and been deleted, the policy for older periodic indices can be deleted.