Reindex from a remote cluster

edit

You can use reindex from remote to migrate indices from your old cluster to a new 6.x cluster. This enables you move to 6.x from a pre-5.6 cluster without interrupting service.

Elasticsearch provides backwards compatibility support that enables indices from the previous major version to be upgraded to the current major version. Skipping a major version means that you must resolve any backward compatibility issues yourself.

Elasticsearch does not support forward compatibility across major versions. For example, you cannot reindex from a 7.x cluster into a 6.x cluster.

To migrate your indices:

  1. Set up a new 6.x cluster alongside your old cluster. Enable it to access your old cluster by adding your old cluster to the reindex.remote.whitelist in elasticsearch.yml:

    reindex.remote.whitelist: oldhost:9200

    The new cluster doesn’t have to start fully-scaled out. As you migrate indices and shift the load to the new cluster, you can add nodes to the new cluster and remove nodes from the old one.

  2. For each index that you need to migrate to the 6.x cluster:

    1. Create a new index in 6.x with the appropriate mappings and settings. Set the refresh_interval to -1 and set number_of_replicas to 0 for faster reindexing.
    2. Reindex from remote to pull documents from the old index into the new 6.x index:

      POST _reindex
      {
        "source": {
          "remote": {
            "host": "http://oldhost:9200",
            "username": "user",
            "password": "pass"
          },
          "index": "source",
          "query": {
            "match": {
              "test": "data"
            }
          }
        },
        "dest": {
          "index": "dest"
        }
      }

      If you run the reindex job in the background by setting wait_for_completion to false, the reindex request returns a task_id you can use to monitor progress of the reindex job with the task API: GET _tasks/TASK_ID.

    3. When the reindex job completes, set the refresh_interval and number_of_replicas to the desired values (the default settings are 30s and 1).
    4. Once replication is complete and the status of the new index is green, you can delete the old index.