Cluster-level shard allocation and routing settings

You can use the following settings to control shard allocation and recovery:

A cluster is balanced when it has an equal number of shards on each node, with all nodes needing equal resources, without having a concentration of shards from any index on any node. Elasticsearch runs an automatic process called rebalancing which moves shards between the nodes in your cluster to improve its balance. Rebalancing obeys all other shard allocation rules such as allocation filtering and forced awareness which may prevent it from completely balancing the cluster. In that case, rebalancing strives to achieve the most balanced cluster possible within the rules you have configured. If you are using data tiers then Elasticsearch automatically applies allocation filtering rules to place each shard within the appropriate tier. These rules mean that the balancer works independently within each tier.

You can use the following settings to control the rebalancing of shards across the cluster:

Rebalancing works by computing a weight for each node based on its allocation of shards, and then moving shards between nodes to reduce the weight of the heavier nodes and increase the weight of the lighter ones. The cluster is balanced when there is no possible shard movement that can bring the weight of any node closer to the weight of any other node by more than a configurable threshold.

The weight of a node depends on the number of shards it holds and on the total estimated resource usage of those shards expressed in terms of the size of the shard on disk and the number of threads needed to support write traffic to the shard. Elasticsearch estimates the resource usage of shards belonging to data streams when they are created by a rollover. The estimated disk size of the new shard is the mean size of the other shards in the data stream. The estimated write load of the new shard is a weighted average of the actual write loads of recent shards in the data stream. Shards that do not belong to the write index of a data stream have an estimated write load of zero.

The following settings control how Elasticsearch combines these values into an overall measure of each node’s weight.

The disk-based shard allocator ensures that all nodes have enough disk space without performing more shard movements than necessary. It allocates shards based on a pair of thresholds known as the low watermark and the high watermark. Its primary goal is to ensure that no node exceeds the high watermark, or at least that any such overage is only temporary. If a node exceeds the high watermark then Elasticsearch will solve this by moving some of its shards onto other nodes in the cluster.

The allocator also tries to keep nodes clear of the high watermark by forbidding the allocation of more shards to a node that exceeds the low watermark. Importantly, if all of your nodes have exceeded the low watermark then no new shards can be allocated and Elasticsearch will not be able to move any shards between nodes in order to keep the disk usage below the high watermark. You must ensure that your cluster has enough disk space in total and that there are always some nodes below the low watermark.

Shard movements triggered by the disk-based shard allocator must also satisfy all other shard allocation rules such as allocation filtering and forced awareness. If these rules are too strict then they can also prevent the shard movements needed to keep the nodes' disk usage under control. If you are using data tiers then Elasticsearch automatically configures allocation filtering rules to place shards within the appropriate tier, which means that the disk-based shard allocator works independently within each tier.

If a node is filling up its disk faster than Elasticsearch can move shards elsewhere then there is a risk that the disk will completely fill up. To prevent this, as a last resort, once the disk usage reaches the flood-stage watermark Elasticsearch will block writes to indices with a shard on the affected node. It will also continue to move shards onto the other nodes in the cluster. When disk usage on the affected node drops below the high watermark, Elasticsearch automatically removes the write block. Refer to Fix watermark errors to resolve persistent watermark errors.

You can use the following settings to control disk-based allocation:

An example of updating the low watermark to at least 100 gigabytes free, a high watermark of at least 50 gigabytes free, and a flood stage watermark of 10 gigabytes free, and updating the information about the cluster every minute:

resp = client.cluster.put_settings(
    persistent={
        "cluster.routing.allocation.disk.watermark.low": "100gb",
        "cluster.routing.allocation.disk.watermark.high": "50gb",
        "cluster.routing.allocation.disk.watermark.flood_stage": "10gb",
        "cluster.info.update.interval": "1m"
    },
)
print(resp)

response = client.cluster.put_settings(
  body: {
    persistent: {
      'cluster.routing.allocation.disk.watermark.low' => '100gb',
      'cluster.routing.allocation.disk.watermark.high' => '50gb',
      'cluster.routing.allocation.disk.watermark.flood_stage' => '10gb',
      'cluster.info.update.interval' => '1m'
    }
  }
)
puts response

const response = await client.cluster.putSettings({
  persistent: {
    "cluster.routing.allocation.disk.watermark.low": "100gb",
    "cluster.routing.allocation.disk.watermark.high": "50gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "10gb",
    "cluster.info.update.interval": "1m",
  },
});
console.log(response);

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "100gb",
    "cluster.routing.allocation.disk.watermark.high": "50gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "10gb",
    "cluster.info.update.interval": "1m"
  }
}

Concerning the max headroom settings for the watermarks, please note that these apply only in the case that the watermark settings are percentages/ratios. The aim of a max headroom value is to cap the required free disk space before hitting the respective watermark. This is especially useful for servers with larger disks, where a percentage/ratio watermark could translate to a big free disk space requirement, and the max headroom can be used to cap the required free disk space amount. As an example, let us take the default settings for the flood watermark. It has a 95% default value, and the flood max headroom setting has a default value of 100GB. This means that:

Finally, the max headroom settings have their default values only if their respective watermark settings are not explicitly set (thus, they have their default percentage values). If watermarks are explicitly set, then the max headroom settings do not have their default values, and would need to be explicitly set if they are desired.

You can use custom node attributes as awareness attributes to enable Elasticsearch to take your physical hardware configuration into account when allocating shards. If Elasticsearch knows which nodes are on the same physical server, in the same rack, or in the same zone, it can distribute the primary shard and its replica shards to minimize the risk of losing all shard copies in the event of a failure.

When shard allocation awareness is enabled with the dynamic cluster.routing.allocation.awareness.attributes setting, shards are only allocated to nodes that have values set for the specified awareness attributes. If you use multiple awareness attributes, Elasticsearch considers each attribute separately when allocating shards.

Enabling shard allocation awareness

edit

To enable shard allocation awareness:

1. Specify the location of each node with a custom node attribute. For example, if you want Elasticsearch to distribute shards across different racks, you might use an awareness attribute called rack_id.

   You can set custom attributes in two ways:

   • By editing the elasticsearch.yml config file:

     node.attr.rack_id: rack_one

   • Using the -E command line argument when you start a node:

     ./bin/elasticsearch -Enode.attr.rack_id=rack_one

2. Tell Elasticsearch to take one or more awareness attributes into account when allocating shards by setting cluster.routing.allocation.awareness.attributes in every master-eligible node’s elasticsearch.yml config file.

   cluster.routing.allocation.awareness.attributes: rack_id 

   Specify multiple attributes as a comma-separated list.

   You can also use the cluster-update-settings API to set or update a cluster’s awareness attributes:

   resp = client.cluster.put_settings(
       persistent={
           "cluster.routing.allocation.awareness.attributes": "rack_id"
       },
   )
   print(resp)

   response = client.cluster.put_settings(
     body: {
       persistent: {
         'cluster.routing.allocation.awareness.attributes' => 'rack_id'
       }
     }
   )
   puts response

   const response = await client.cluster.putSettings({
     persistent: {
       "cluster.routing.allocation.awareness.attributes": "rack_id",
     },
   });
   console.log(response);

   PUT /_cluster/settings
   {
     "persistent" : {
       "cluster.routing.allocation.awareness.attributes" : "rack_id"
     }
   }

With this example configuration, if you start two nodes with node.attr.rack_id set to rack_one and create an index with 5 primary shards and 1 replica of each primary, all primaries and replicas are allocated across the two node.

Figure 1. All primaries and replicas allocated across two nodes in the same rack

If you add two nodes with node.attr.rack_id set to rack_two, Elasticsearch moves shards to the new nodes, ensuring (if possible) that no two copies of the same shard are in the same rack.

Figure 2. Primaries and replicas allocated across four nodes in two racks, with no two copies of the same shard in the same rack

If rack_two fails and takes down both its nodes, by default Elasticsearch allocates the lost shard copies to nodes in rack_one. To prevent multiple copies of a particular shard from being allocated in the same location, you can enable forced awareness.

cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: zone1,zone2 

You can use cluster-level shard allocation filters to control where Elasticsearch allocates shards from any index. These cluster wide filters are applied in conjunction with per-index allocation filtering and allocation awareness.

Shard allocation filters can be based on custom node attributes or the built-in _name, _host_ip, _publish_ip, _ip, _host, _id and _tier attributes.

The cluster.routing.allocation settings are dynamic, enabling live indices to be moved from one set of nodes to another. Shards are only relocated if it is possible to do so without breaking another routing constraint, such as never allocating a primary and replica shard on the same node.

The most common use case for cluster-level shard allocation filtering is when you want to decommission a node. To move shards off of a node prior to shutting it down, you could create a filter that excludes the node by its IP address:

resp = client.cluster.put_settings(
    persistent={
        "cluster.routing.allocation.exclude._ip": "10.0.0.1"
    },
)
print(resp)

response = client.cluster.put_settings(
  body: {
    persistent: {
      'cluster.routing.allocation.exclude._ip' => '10.0.0.1'
    }
  }
)
puts response

const response = await client.cluster.putSettings({
  persistent: {
    "cluster.routing.allocation.exclude._ip": "10.0.0.1",
  },
});
console.log(response);

PUT _cluster/settings
{
  "persistent" : {
    "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
  }
}

resp = client.cluster.put_settings(
    persistent={
        "cluster.routing.allocation.exclude._ip": "192.168.2.*"
    },
)
print(resp)

response = client.cluster.put_settings(
  body: {
    persistent: {
      'cluster.routing.allocation.exclude._ip' => '192.168.2.*'
    }
  }
)
puts response

const response = await client.cluster.putSettings({
  persistent: {
    "cluster.routing.allocation.exclude._ip": "192.168.2.*",
  },
});
console.log(response);

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.exclude._ip": "192.168.2.*"
  }
}