IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Managing existing periodic indices with ILM Getting started with snapshot lifecycle management »
Elastic Docs ›Elasticsearch Guide [7.5] ›Managing the index lifecycle ›Using ILM with existing indices

Reindexing via ILM

edit

Reindexing via ILM

edit

The examples in this section assume daily indices as set up in the previous section.

In some cases, it may be useful to reindex data into ILM-managed indices. This is more complex than simply attaching policies to existing indices as described in the previous section, and requires pausing indexing during the reindexing process. However, this technique may be useful in cases where periodic indices were created with very small amounts of data leading to excessive shard counts, or for indices which grow steadily over time, but have not been broken up into time-series indices leading to shards which are much too large, situations that cause significant performance problems.

Before getting started with reindexing data, the new index structure should be set up. For this section, we’ll be using the same setup described in {ilm-imit} with existing indices.

First, we’ll set up a policy with rollover, and can include any additional phases required. For simplicity, we’ll just use rollover:

PUT _ilm/policy/sample_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_size": "50G"
          }
        }
      }
    }
  }
}

And now we’ll update the index template for our indices to include the relevant ILM settings:

PUT _template/mylogs_template
{
  "index_patterns": [
    "ilm-mylogs-*" 
  ],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index": {
      "lifecycle": {
        "name": "mylogs_condensed_policy", 
        "rollover_alias": "mylogs" 
      }
    }
  },
  "mappings": {
    "properties": {
      "message": {
        "type": "text"
      },
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

The new index pattern has a prefix compared to the old one, this will make it easier to reindex later

The name of the policy we defined above

The name of the alias we’ll use to write to and query

And create the first index with the alias specified in the rollover_alias setting in the index template:

PUT ilm-mylogs-000001
{
  "aliases": {
    "mylogs": {
      "is_write_index": true
    }
  }
}

All new documents should be indexed via the mylogs alias at this point. Adding new data to the old indices during the reindexing process can cause data to be added to the old indices, but not be reindexed into the new indices.

If you do not want to mix new data and old data in the new ILM-managed indices, indexing of new data should be paused entirely while the reindex completes. Mixing old and new data within one index is safe, but keep in mind that the indices with mixed data should be retained in their entirety until you are ready to delete both the old and new data.

By default, ILM only checks rollover conditions every 10 minutes. Under normal indexing load, this usually works well, but during reindexing, indices can grow very, very quickly. We’ll need to set the poll interval to something shorter to ensure that the new indices don’t grow too large while waiting for the rollover check:

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "1m" 
  }
}

This tells ILM to check for rollover conditions every minute

We’re now ready to reindex our data using the reindex API. If you have a timestamp or date field in your documents, as in this example, it may be useful to specify that the documents should be sorted by that field - this will mean that all documents in ilm-mylogs-000001 come before all documents in ilm-mylogs-000002, and so on. However, if this is not a requirement, omitting the sort will allow the data to be reindexed more quickly.

If your data uses document IDs generated by means other than Elasticsearch’s automatic ID generation, you may need to do additional processing to ensure that the document IDs don’t conflict during the reindex, as documents will retain their original IDs. One way to do this is to use a script in the reindex call to append the original index name to the document ID.

POST _reindex
{
  "source": {
    "index": "mylogs-*", 
    "sort": { "@timestamp": "desc" }
  },
  "dest": {
    "index": "mylogs", 
    "op_type": "create" 
  }
}

This index pattern matches our existing indices. Using the prefix for the new indices makes using this index pattern much easier.

The alias set up above

This option will cause the reindex to abort if it encounters multiple documents with the same ID. This is optional, but recommended to prevent accidentally overwriting documents if two documents from different indices have the same ID.

Once this completes, indexing new data can be resumed, as long as all new documents are indexed into the alias used above. All data, existing and new, can be queried using that alias as well. We should also be sure to set the ILM poll interval back to its default value, because keeping it set too low can cause unnecessary load on the current master node:

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": null
  }
}

All of the reindexed data should now be accessible via the alias set up above, in this case mylogs. Once you have verified that all the data has been reindexed and is available in the new indices, the existing indices can be safely removed.

« Managing existing periodic indices with ILM Getting started with snapshot lifecycle management »