IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Configure a lifecycle policy Resolve lifecycle policy execution errors »
Elastic Docs ›Elasticsearch Guide [7.10] ›ILM: Manage the index lifecycle

Migrate index allocation filters to node roles

edit

Migrate index allocation filters to node roles

edit

If you currently use custom node attributes and attribute-based allocation filters to move indices through data tiers in a hot-warm-cold architecture, we recommend that you switch to using the built-in node roles and automatic data tier allocation. Using node roles enables ILM to automatically move indices between data tiers.

While we recommend relying on automatic data tier allocation to manage your data in a hot-warm-cold architecture, you can still use attribute-based allocation filters to control shard allocation for other purposes.

To switch to using node roles:

  1. Assign data nodes to the appropriate data tier.
  2. Remove the attribute-based allocation settings from your index lifecycle management policy.
  3. Stop setting the custom hot attribute on new indices.
  4. Update existing indices to set a tier preference.

Assign data nodes to a data tier

edit

Configure the appropriate roles for each data node to assign it to one or more data tiers: data_hot, data_content, data_warm, data_cold, or data_frozen. A node can also have other roles. By default, new nodes are configured with all roles.

When you add a data tier to an Elasticsearch Service deployment, one or more nodes are automatically configured with the corresponding role. To explicitly change the role of a node in an Elasticsearch Service deployment, use the Update deployment API. Replace the node’s node_type configuration with the appropriate node_roles. For example, the following configuration adds the node to the hot and content tiers, and enables it to act as an ingest node, remote, and transform node.

"node_roles": [
  "data_hot",
  "data_content",
  "ingest",
  "remote_cluster_client",
  "transform"
],

If you are directly managing your own cluster, configure the appropriate roles for each node in elasticsearch.yml. For example, the following setting configures a node to be a data-only node in the hot and content tiers.

node.roles [ data_hot, data_content ]

Remove custom allocation settings from existing ILM policies

edit

Update the allocate action for each lifecycle phase to remove the attribute-based allocation settings. This enables ILM to inject the migrate action into each phase to automatically transition the indices through the data tiers.

If the allocate action does not set the number of replicas, remove the allocate action entirely. (An empty allocate action is invalid.)

The policy must specify the corresponding phase for each data tier in your architecture. Each phase must be present so ILM can inject the migrate action to move indices through the data tiers. If you don’t need to perform any other actions, the phase can be empty. For example, if you enable the warm and cold data tiers for a deployment, your policy must include the hot, warm, and cold phases.

Stop setting the custom hot attribute on new indices

edit

When you create a data stream, its first backing index is now automatically assigned to data_hot nodes. Similarly, when you directly create an index, it is automatically assigned to data_content nodes.

On Elasticsearch Service deployments, remove the cloud-hot-warm-allocation-0 index template that set the hot shard allocation attribute on all indices.

DELETE _template/.cloud-hot-warm-allocation-0

If you’re using a custom index template, update it to remove the attribute-based allocation filters you used to assign new indices to the hot tier.

Set a tier preference for existing indices.

edit

ILM automatically transitions managed indices through the available data tiers by automatically injecting a migrate action into each phase.

To enable ILM to move an existing managed index through the data tiers, update the index settings to:

  1. Remove the custom allocation filter by setting it to null.
  2. Set the tier preference.

For example, if your old template set the data attribute to hot to allocate shards to the hot tier, set the data attribute to null and set the _tier_preference to data_hot.

PUT my-index/_settings
{
  "index.routing.allocation.require.data": null,
  "index.routing.allocation.include._tier_preference": "data_hot"
}

For indices that have already transitioned out of the hot phase, the tier preference should include the appropriate fallback tiers to ensure index shards can be allocated if the preferred tier is unavailable. For example, specify the hot tier as the fallback for indices already in the warm phase.

PUT my-index/_settings
{
  "index.routing.allocation.require.data": null,
  "index.routing.allocation.include._tier_preference": "data_warm,data_hot"
}

If an index is already in the cold phase, include the cold, warm, and hot tiers.

« Configure a lifecycle policy Resolve lifecycle policy execution errors »