WARNING: Version 6.2 of Elasticsearch has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

› ›

Split Index

edit

Split Index

edit

The split index API allows you to split an existing index into a new index, where each original primary shard is split into two or more primary shards in the new index.

The _split API requires the source index to be created with a specific number_of_routing_shards in order to be split in the future. This requirement has been removed in Elasticsearch 7.0.

The number of times the index can be split (and the number of shards that each original shard can be split into) is determined by the index.number_of_routing_shards setting. The number of routing shards specifies the hashing space that is used internally to distribute documents across shards with consistent hashing. For instance, a 5 shard index with number_of_routing_shards set to 30 (5 x 2 x 3) could be split by a factor of 2 or 3. In other words, it could be split as follows:

5 → 10 → 30 (split by 2, then by 3)
5 → 15 → 30 (split by 3, then by 2)
5 → 30 (split by 6)

Splitting works as follows:

First, it creates a new target index with the same definition as the source index, but with a larger number of primary shards.
Then it hard-links segments from the source index into the target index. (If the file system doesn’t support hard-linking, then all segments are copied into the new index, which is a much more time consuming process.)
Once the low level files are created all documents will be hashed again to delete documents that belong to a different shard.
Finally, it recovers the target index as though it were a closed index which had just been re-opened.

Preparing an index for splitting

edit

Create an index with a routing shards factor:

PUT my_source_index
{
    "settings": {
        "index.number_of_shards" : 1,
        "index.number_of_routing_shards" : 2 
    }
}

Allows to split the index into two shards or in other words, it allows for a single split operation.

In order to split an index, the index must be marked as read-only, and have health green.

This can be achieved with the following request:

PUT /my_source_index/_settings
{
  "settings": {
    "index.blocks.write": true 
  }
}

Prevents write operations to this index while still allowing metadata changes like deleting the index.

Splitting an index

edit

To split my_source_index into a new index called my_target_index, issue the following request:

POST my_source_index/_split/my_target_index
{
  "settings": {
    "index.number_of_shards": 2
  }
}

The above request returns immediately once the target index has been added to the cluster state — it doesn’t wait for the split operation to start.

Indices can only be split if they satisfy the following requirements:

the target index must not exist
The index must have less primary shards than the target index.
The number of primary shards in the target index must be a factor of the number of primary shards in the source index.
The node handling the split process must have sufficient free disk space to accommodate a second copy of the existing index.

The _split API is similar to the create index API and accepts settings and aliases parameters for the target index:

POST my_source_index/_split/my_target_index
{
  "settings": {
    "index.number_of_shards": 5 
  },
  "aliases": {
    "my_search_indices": {}
  }
}

The number of shards in the target index. This must be a factor of the number of shards in the source index.

Mappings may not be specified in the _split request, and all index.analysis.* and index.similarity.* settings will be overwritten with the settings from the source index.

Monitoring the split process

edit

The split process can be monitored with the _cat recovery API, or the cluster health API can be used to wait until all primary shards have been allocated by setting the wait_for_status parameter to yellow.

The _split API returns as soon as the target index has been added to the cluster state, before any shards have been allocated. At this point, all shards are in the state unassigned. If, for any reason, the target index can’t be allocated, its primary shard will remain unassigned until it can be allocated on that node.

Once the primary shard is allocated, it moves to state initializing, and the split process begins. When the split operation completes, the shard will become active. At that point, Elasticsearch will try to allocate any replicas and may decide to relocate the primary shard to another node.

Wait For Active Shards

edit

Because the split operation creates a new index to split the shards to, the wait for active shards setting on index creation applies to the split index action as well.

« Shrink Index Rollover Index »