Cluster-level shard allocation and routing settings
editCluster-level shard allocation and routing settings
editShard allocation is the process of allocating shards to nodes. This can happen during initial recovery, replica allocation, rebalancing, or when nodes are added or removed.
One of the main roles of the master is to decide which shards to allocate to which nodes, and when to move shards between nodes in order to rebalance the cluster.
There are a number of settings available to control the shard allocation process:
- Cluster-level shard allocation settings control allocation and rebalancing operations.
- Disk-based shard allocation settings explains how Elasticsearch takes available disk space into account, and the related settings.
- Shard allocation awareness and Forced awareness control how shards can be distributed across different racks or availability zones.
- Cluster-level shard allocation filtering allows certain nodes or groups of nodes excluded from allocation so that they can be decommissioned.
Besides these, there are a few other miscellaneous cluster-level settings.
All of these settings are dynamic and can be updated on a live cluster with the cluster-update-settings API.
Cluster-level shard allocation settings
editThe following dynamic settings may be used to control shard allocation and recovery:
-
cluster.routing.allocation.enable
-
Enable or disable allocation for specific kinds of shards:
-
all
- (default) Allows shard allocation for all kinds of shards. -
primaries
- Allows shard allocation only for primary shards. -
new_primaries
- Allows shard allocation only for primary shards for new indices. -
none
- No shard allocations of any kind are allowed for any indices.
This setting does not affect the recovery of local primary shards when restarting a node. A restarted node that has a copy of an unassigned primary shard will recover that primary immediately, assuming that its allocation id matches one of the active allocation ids in the cluster state.
-
-
cluster.routing.allocation.node_concurrent_incoming_recoveries
-
How many concurrent incoming shard recoveries are allowed to happen on a node. Incoming recoveries are the recoveries
where the target shard (most likely the replica unless a shard is relocating) is allocated on the node. Defaults to
2
. -
cluster.routing.allocation.node_concurrent_outgoing_recoveries
-
How many concurrent outgoing shard recoveries are allowed to happen on a node. Outgoing recoveries are the recoveries
where the source shard (most likely the primary unless a shard is relocating) is allocated on the node. Defaults to
2
. -
cluster.routing.allocation.node_concurrent_recoveries
-
A shortcut to set both
cluster.routing.allocation.node_concurrent_incoming_recoveries
andcluster.routing.allocation.node_concurrent_outgoing_recoveries
. -
cluster.routing.allocation.node_initial_primaries_recoveries
-
While the recovery of replicas happens over the network, the recovery of
an unassigned primary after node restart uses data from the local disk.
These should be fast so more initial primary recoveries can happen in
parallel on the same node. Defaults to
4
. -
cluster.routing.allocation.same_shard.host
-
Allows to perform a check to prevent allocation of multiple instances of
the same shard on a single host, based on host name and host address.
Defaults to
false
, meaning that no check is performed by default. This setting only applies if multiple nodes are started on the same machine.
Shard rebalancing settings
editThe following dynamic settings may be used to control the rebalancing of shards across the cluster:
-
cluster.routing.rebalance.enable
-
Enable or disable rebalancing for specific kinds of shards:
-
all
- (default) Allows shard balancing for all kinds of shards. -
primaries
- Allows shard balancing only for primary shards. -
replicas
- Allows shard balancing only for replica shards. -
none
- No shard balancing of any kind are allowed for any indices.
-
-
cluster.routing.allocation.allow_rebalance
-
Specify when shard rebalancing is allowed:
-
always
- Always allow rebalancing. -
indices_primaries_active
- Only when all primaries in the cluster are allocated. -
indices_all_active
- (default) Only when all shards (primaries and replicas) in the cluster are allocated.
-
-
cluster.routing.allocation.cluster_concurrent_rebalance
-
Allow to control how many concurrent shard rebalances are
allowed cluster wide. Defaults to
2
. Note that this setting only controls the number of concurrent shard relocations due to imbalances in the cluster. This setting does not limit shard relocations due to allocation filtering or forced awareness.
Shard balancing heuristics settings
editThe following settings are used together to determine where to place each
shard. The cluster is balanced when no allowed rebalancing operation can bring the weight
of any node closer to the weight of any other node by more than the balance.threshold
.
-
cluster.routing.allocation.balance.shard
-
Defines the weight factor for the total number of shards allocated on a node
(float). Defaults to
0.45f
. Raising this raises the tendency to equalize the number of shards across all nodes in the cluster. -
cluster.routing.allocation.balance.index
-
Defines the weight factor for the number of shards per index allocated
on a specific node (float). Defaults to
0.55f
. Raising this raises the tendency to equalize the number of shards per index across all nodes in the cluster. -
cluster.routing.allocation.balance.threshold
-
Minimal optimization value of operations that should be performed (non
negative float). Defaults to
1.0f
. Raising this will cause the cluster to be less aggressive about optimizing the shard balance.
Regardless of the result of the balancing algorithm, rebalancing might not be allowed due to forced awareness or allocation filtering.
Disk-based shard allocation settings
editElasticsearch considers the available disk space on a node before deciding whether to allocate new shards to that node or to actively relocate shards away from that node.
Below are the settings that can be configured in the elasticsearch.yml
config
file or updated dynamically on a live cluster with the
cluster-update-settings API:
-
cluster.routing.allocation.disk.watermark.low
-
Controls the low watermark for disk usage. It defaults to
85%
, meaning that Elasticsearch will not allocate shards to nodes that have more than 85% disk used. It can also be set to an absolute byte value (like500mb
) to prevent Elasticsearch from allocating shards if less than the specified amount of space is available. This setting has no effect on the primary shards of newly-created indices but will prevent their replicas from being allocated.
-
cluster.routing.allocation.disk.watermark.high
-
Controls the high watermark. It defaults to
90%
, meaning that Elasticsearch will attempt to relocate shards away from a node whose disk usage is above 90%. It can also be set to an absolute byte value (similarly to the low watermark) to relocate shards away from a node if it has less than the specified amount of free space. This setting affects the allocation of all shards, whether previously allocated or not. -
cluster.routing.allocation.disk.watermark.enable_for_single_data_node
-
For a single data node, the default is to disregard disk watermarks when
making an allocation decision. This is deprecated behavior and will be
changed in 8.0. This setting can be set to
true
to enable the disk watermarks for a single data node cluster (will become default in 8.0).
-
cluster.routing.allocation.disk.watermark.flood_stage
-
Controls the flood stage watermark, which defaults to 95%. Elasticsearch enforces a read-only index block (
index.blocks.read_only_allow_delete
) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is a last resort to prevent nodes from running out of disk space. The index block is automatically released when the disk utilization falls below the high watermark.You cannot mix the usage of percentage values and byte values within these settings. Either all values are set to percentage values, or all are set to byte values. This enforcement is so that Elasticsearch can validate that the settings are internally consistent, ensuring that the low disk threshold is less than the high disk threshold, and the high disk threshold is less than the flood stage threshold.
An example of resetting the read-only index block on the
my-index-000001
index:PUT /my-index-000001/_settings { "index.blocks.read_only_allow_delete": null }
-
cluster.info.update.interval
-
How often Elasticsearch should check on disk usage for each node in the
cluster. Defaults to
30s
. -
cluster.routing.allocation.disk.include_relocations
-
[7.5.0]
Deprecated in 7.5.0. Future versions will always account for relocations.
Defaults to
true
, which means that Elasticsearch will take into account shards that are currently being relocated to the target node when computing a node’s disk usage. Taking relocating shards' sizes into account may, however, mean that the disk usage for a node is incorrectly estimated on the high side, since the relocation could be 90% complete and a recently retrieved disk usage would include the total size of the relocating shard as well as the space already used by the running relocation.
Percentage values refer to used disk space, while byte values refer to free disk space. This can be confusing, since it flips the meaning of high and low. For example, it makes sense to set the low watermark to 10gb and the high watermark to 5gb, but not the other way around.
An example of updating the low watermark to at least 100 gigabytes free, a high watermark of at least 50 gigabytes free, and a flood stage watermark of 10 gigabytes free, and updating the information about the cluster every minute:
PUT _cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.low": "100gb", "cluster.routing.allocation.disk.watermark.high": "50gb", "cluster.routing.allocation.disk.watermark.flood_stage": "10gb", "cluster.info.update.interval": "1m" } }
Shard allocation awareness
editYou can use custom node attributes as awareness attributes to enable Elasticsearch to take your physical hardware configuration into account when allocating shards. If Elasticsearch knows which nodes are on the same physical server, in the same rack, or in the same zone, it can distribute the primary shard and its replica shards to minimise the risk of losing all shard copies in the event of a failure.
When shard allocation awareness is enabled with the
cluster.routing.allocation.awareness.attributes
setting, shards are only
allocated to nodes that have values set for the specified awareness
attributes. If you use multiple awareness attributes, Elasticsearch considers
each attribute separately when allocating shards.
The allocation awareness settings can be configured in
elasticsearch.yml
and updated dynamically with the
cluster-update-settings API.
By default Elasticsearch uses adaptive replica selection
to route search or GET requests. However, with the presence of allocation awareness
attributes Elasticsearch will prefer using shards in the same location (with the same
awareness attribute values) to process these requests. This behavior can be
disabled by specifying export ES_JAVA_OPTS="$ES_JAVA_OPTS -Des.search.ignore_awareness_attributes=true"
system property on every node that is part of the cluster.
The number of attribute values determines how many shard copies are allocated in each location. If the number of nodes in each location is unbalanced and there are a lot of replicas, replica shards might be left unassigned.
Enabling shard allocation awareness
editTo enable shard allocation awareness:
-
Specify the location of each node with a custom node attribute. For example, if you want Elasticsearch to distribute shards across different racks, you might set an awareness attribute called
rack_id
in each node’selasticsearch.yml
config file.node.attr.rack_id: rack_one
You can also set custom attributes when you start a node:
`./bin/elasticsearch -Enode.attr.rack_id=rack_one`
-
Tell Elasticsearch to take one or more awareness attributes into account when allocating shards by setting
cluster.routing.allocation.awareness.attributes
in every master-eligible node’selasticsearch.yml
config file.You can also use the cluster-update-settings API to set or update a cluster’s awareness attributes.
With this example configuration, if you start two nodes with
node.attr.rack_id
set to rack_one
and create an index with 5 primary
shards and 1 replica of each primary, all primaries and replicas are
allocated across the two nodes.
If you add two nodes with node.attr.rack_id
set to rack_two
,
Elasticsearch moves shards to the new nodes, ensuring (if possible)
that no two copies of the same shard are in the same rack.
If rack_two
fails and takes down both its nodes, by default Elasticsearch
allocates the lost shard copies to nodes in rack_one
. To prevent multiple
copies of a particular shard from being allocated in the same location, you can
enable forced awareness.
Forced awareness
editBy default, if one location fails, Elasticsearch assigns all of the missing replica shards to the remaining locations. While you might have sufficient resources across all locations to host your primary and replica shards, a single location might be unable to host ALL of the shards.
To prevent a single location from being overloaded in the event of a failure,
you can set cluster.routing.allocation.awareness.force
so no replicas are
allocated until nodes are available in another location.
For example, if you have an awareness attribute called zone
and configure nodes
in zone1
and zone2
, you can use forced awareness to prevent Elasticsearch
from allocating replicas if only one zone is available:
cluster.routing.allocation.awareness.attributes: zone cluster.routing.allocation.awareness.force.zone.values: zone1,zone2
With this example configuration, if you start two nodes with node.attr.zone
set
to zone1
and create an index with 5 shards and 1 replica, Elasticsearch creates
the index and allocates the 5 primary shards but no replicas. Replicas are
only allocated once nodes with node.attr.zone
set to zone2
are available.
Cluster-level shard allocation filtering
editYou can use cluster-level shard allocation filters to control where Elasticsearch allocates shards from any index. These cluster wide filters are applied in conjunction with per-index allocation filtering and allocation awareness.
Shard allocation filters can be based on custom node attributes or the built-in
_name
, _host_ip
, _publish_ip
, _ip
, _host
and _id
attributes.
The cluster.routing.allocation
settings are dynamic, enabling live indices to
be moved from one set of nodes to another. Shards are only relocated if it is
possible to do so without breaking another routing constraint, such as never
allocating a primary and replica shard on the same node.
The most common use case for cluster-level shard allocation filtering is when you want to decommission a node. To move shards off of a node prior to shutting it down, you could create a filter that excludes the node by its IP address:
PUT _cluster/settings { "transient" : { "cluster.routing.allocation.exclude._ip" : "10.0.0.1" } }
Cluster routing settings
edit-
cluster.routing.allocation.include.{attribute}
-
Allocate shards to a node whose
{attribute}
has at least one of the comma-separated values. -
cluster.routing.allocation.require.{attribute}
-
Only allocate shards to a node whose
{attribute}
has all of the comma-separated values. -
cluster.routing.allocation.exclude.{attribute}
-
Do not allocate shards to a node whose
{attribute}
has any of the comma-separated values.
The cluster allocation settings support the following built-in attributes:
|
Match nodes by node name |
|
Match nodes by host IP address (IP associated with hostname) |
|
Match nodes by publish IP address |
|
Match either |
|
Match nodes by hostname |
|
Match nodes by node id |
You can use wildcards when specifying attribute values, for example:
PUT _cluster/settings { "transient": { "cluster.routing.allocation.exclude._ip": "192.168.2.*" } }
Miscellaneous cluster settings
editMetadata
editAn entire cluster may be set to read-only with the following dynamic setting:
-
cluster.blocks.read_only
- Make the whole cluster read only (indices do not accept write operations), metadata is not allowed to be modified (create or delete indices).
-
cluster.blocks.read_only_allow_delete
-
Identical to
cluster.blocks.read_only
but allows to delete indices to free up resources.
Don’t rely on this setting to prevent changes to your cluster. Any user with access to the cluster-update-settings API can make the cluster read-write again.
Cluster shard limit
editThere is a soft limit on the number of shards in a cluster, based on the number of nodes in the cluster. This is intended to prevent operations which may unintentionally destabilize the cluster.
This limit is intended as a safety net, not a sizing recommendation. The exact number of shards your cluster can safely support depends on your hardware configuration and workload, but should remain well below this limit in almost all cases, as the default limit is set quite high.
If an operation, such as creating a new index, restoring a snapshot of an index, or opening a closed index would lead to the number of shards in the cluster going over this limit, the operation will fail with an error indicating the shard limit.
If the cluster is already over the limit, due to changes in node membership or setting changes, all operations that create or open indices will fail until either the limit is increased as described below, or some indices are closed or deleted to bring the number of shards below the limit.
Replicas count towards this limit, but closed indexes do not. An index with 5 primary shards and 2 replicas will be counted as 15 shards. Any closed index is counted as 0, no matter how many shards and replicas it contains.
The limit defaults to 1,000 shards per data node, and can be dynamically adjusted using the following property:
-
cluster.max_shards_per_node
- Controls the number of shards allowed in the cluster per data node.
For example, a 3-node cluster with the default setting would allow 3,000 shards total, across all open indexes. If the above setting is changed to 500, then the cluster would allow 1,500 shards total.
If there are no data nodes in the cluster, the limit will not be enforced. This allows the creation of indices during cluster creation if dedicated master nodes are set up before data nodes.
User-defined cluster metadata
editUser-defined metadata can be stored and retrieved using the Cluster Settings API.
This can be used to store arbitrary, infrequently-changing data about the cluster
without the need to create an index to store it. This data may be stored using
any key prefixed with cluster.metadata.
. For example, to store the email
address of the administrator of a cluster under the key cluster.metadata.administrator
,
issue this request:
PUT /_cluster/settings { "persistent": { "cluster.metadata.administrator": "sysadmin@example.com" } }
User-defined cluster metadata is not intended to store sensitive or confidential information. Any information stored in user-defined cluster metadata will be viewable by anyone with access to the Cluster Get Settings API, and is recorded in the Elasticsearch logs.
Index tombstones
editThe cluster state maintains index tombstones to explicitly denote indices that have been deleted. The number of tombstones maintained in the cluster state is controlled by the following property, which cannot be updated dynamically:
-
cluster.indices.tombstones.size
-
Index tombstones prevent nodes that are not part of the cluster when a delete
occurs from joining the cluster and reimporting the index as though the delete
was never issued. To keep the cluster state from growing huge we only keep the
last
cluster.indices.tombstones.size
deletes, which defaults to 500. You can increase it if you expect nodes to be absent from the cluster and miss more than 500 deletes. We think that is rare, thus the default. Tombstones don’t take up much space, but we also think that a number like 50,000 is probably too big.
Logger
editThe settings which control logging can be updated dynamically with the
logger.
prefix. For instance, to increase the logging level of the
indices.recovery
module to DEBUG
, issue this request:
PUT /_cluster/settings { "transient": { "logger.org.elasticsearch.indices.recovery": "DEBUG" } }
Persistent tasks allocation
editPlugins can create a kind of tasks called persistent tasks. Those tasks are usually long-lived tasks and are stored in the cluster state, allowing the tasks to be revived after a full cluster restart.
Every time a persistent task is created, the master node takes care of assigning the task to a node of the cluster, and the assigned node will then pick up the task and execute it locally. The process of assigning persistent tasks to nodes is controlled by the following properties, which can be updated dynamically:
-
cluster.persistent_tasks.allocation.enable
-
Enable or disable allocation for persistent tasks:
-
all
- (default) Allows persistent tasks to be assigned to nodes -
none
- No allocations are allowed for any type of persistent task
This setting does not affect the persistent tasks that are already being executed. Only newly created persistent tasks, or tasks that must be reassigned (after a node left the cluster, for example), are impacted by this setting.
-
-
cluster.persistent_tasks.allocation.recheck_interval
- The master node will automatically check whether persistent tasks need to be assigned when the cluster state changes significantly. However, there may be other factors, such as memory usage, that affect whether persistent tasks can be assigned to nodes but do not cause the cluster state to change. This setting controls how often assignment checks are performed to react to these factors. The default is 30 seconds. The minimum permitted value is 10 seconds.