Shrink

edit

Phases allowed: hot, warm.

Blocks writes on a source index and shrinks it into a new index with fewer primary shards. The name of the resulting index is shrink-<random-uuid>-<original-index-name>. This action corresponds to the shrink API.

After the shrink action, any aliases that pointed to the source index point to the new shrunken index. If ILM performs the shrink action on a backing index for a data stream, the shrunken index replaces the source index in the stream. You cannot perform the shrink action on a write index.

To use the shrink action in the hot phase, the rollover action must be present. If no rollover action is configured, ILM will reject the policy.

The shrink action will unset the index’s index.routing.allocation.total_shards_per_node setting, meaning that there will be no limit. This is to ensure that all shards of the index can be copied to a single node. This setting change will persist on the index even after the step completes.

If the shrink action is used on a follower index, policy execution waits until the leader index rolls over (or is otherwise marked complete), then converts the follower index into a regular index with the unfollow action before performing the shrink operation.

Shrink options

edit
number_of_shards
(Optional, integer) Number of shards to shrink to. Must be a factor of the number of shards in the source index. This parameter conflicts with max_primary_shard_size, only one of them may be set.
max_primary_shard_size
(Optional, byte units) The max primary shard size for the target index. Used to find the optimum number of shards for the target index. When this parameter is set, each shard’s storage in the target index will not be greater than the parameter. The shards count of the target index will still be a factor of the source index’s shards count, but if the parameter is less than the single shard size in the source index, the shards count for the target index will be equal to the source index’s shards count. For example, when this parameter is set to 50gb, if the source index has 60 primary shards with totaling 100gb, then the target index will have 2 primary shards, with each shard size of 50gb; if the source index has 60 primary shards with totaling 1000gb, then the target index will have 20 primary shards; if the source index has 60 primary shards with totaling 4000gb, then the target index will still have 60 primary shards. This parameter conflicts with number_of_shards in the settings, only one of them may be set.
allow_write_after_shrink
(Optional, boolean) If true, the shrunken index is made writable by removing the write block. Defaults to false.

Example

edit

Set the number of shards of the new shrunken index explicitly

edit
response = client.ilm.put_lifecycle(
  policy: 'my_policy',
  body: {
    policy: {
      phases: {
        warm: {
          actions: {
            shrink: {
              number_of_shards: 1
            }
          }
        }
      }
    }
  }
)
puts response
PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "warm": {
        "actions": {
          "shrink" : {
            "number_of_shards": 1
          }
        }
      }
    }
  }
}

Calculate the optimal number of primary shards for a shrunken index

edit

The following policy uses the max_primary_shard_size parameter to automatically calculate the new shrunken index’s primary shard count based on the source index’s storage size.

response = client.ilm.put_lifecycle(
  policy: 'my_policy',
  body: {
    policy: {
      phases: {
        warm: {
          actions: {
            shrink: {
              max_primary_shard_size: '50gb'
            }
          }
        }
      }
    }
  }
)
puts response
PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "warm": {
        "actions": {
          "shrink" : {
            "max_primary_shard_size": "50gb"
          }
        }
      }
    }
  }
}

Shard allocation for shrink

edit

During a shrink action, ILM allocates the source index’s primary shards to one node. After shrinking the index, ILM reallocates the shrunken index’s shards to the appropriate nodes based on your allocation rules.

These allocation steps can fail for several reasons, including:

  • A node is removed during the shrink action.
  • No node has enough disk space to host the source index’s shards.
  • Elasticsearch cannot reallocate the shrunken index due to conflicting allocation rules.

When one of the allocation steps fails, ILM waits for the period set in index.lifecycle.step.wait_time_threshold, which defaults to 12 hours. This threshold period lets the cluster resolve any issues causing the allocation failure.

If the threshold period passes and ILM has not yet shrunk the index, ILM attempts to allocate the source index’s primary shards to another node. If ILM shrunk the index but could not reallocate the shrunken index’s shards during the threshold period, ILM deletes the shrunken index and re-attempts the entire shrink action.