Total number of shards per node has been reached
editTotal number of shards per node has been reached
editElasticsearch tries to take advantage of all the available resources by distributing data (index shards) amongst the cluster nodes.
Users might want to influence this data distribution by configuring the
cluster.routing.allocation.total_shards_per_node
system setting to restrict the number of shards that can be hosted on a single
node in the system, regardless of the index.
Various configurations limiting how many shards can be hosted on a single node
can lead to shards being unassigned due to the cluster not having enough nodes to
satisfy the configuration.
In order to fix this follow the next steps:
In order to get the shards assigned we’ll need to increase the number of shards
that can be collocated on a node in the cluster.
We’ll achieve this by inspecting the system-wide cluster.routing.allocation.total_shards_per_node
cluster setting and increasing the configured value.
Use Kibana
- Log in to the Elastic Cloud console.
-
On the Elasticsearch Service panel, click the name of your deployment.
If the name of your deployment is disabled your Kibana instances might be unhealthy, in which case please contact Elastic Support. If your deployment doesn’t include Kibana, all you need to do is enable it first.
-
Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to Dev Tools > Console.
-
Inspect the
cluster.routing.allocation.total_shards_per_node
cluster setting:resp = client.cluster.get_settings( flat_settings=True, ) print(resp)
response = client.cluster.get_settings( flat_settings: true ) puts response
const response = await client.cluster.getSettings({ flat_settings: "true", }); console.log(response);
GET /_cluster/settings?flat_settings
The response will look like this:
-
Increase the value for the total number of shards that can be assigned on one node to a higher value:
resp = client.cluster.put_settings( persistent={ "cluster.routing.allocation.total_shards_per_node": 400 }, ) print(resp)
response = client.cluster.put_settings( body: { persistent: { 'cluster.routing.allocation.total_shards_per_node' => 400 } } ) puts response
const response = await client.cluster.putSettings({ persistent: { "cluster.routing.allocation.total_shards_per_node": 400, }, }); console.log(response);
In order to get the shards assigned you can add more nodes to your Elasticsearch cluster and assign the index’s target tier node role to the new nodes.
To inspect which tier is an index targeting for assignment, use the get index setting
API to retrieve the configured value for the index.routing.allocation.include._tier_preference
setting:
resp = client.indices.get_settings( index="my-index-000001", name="index.routing.allocation.include._tier_preference", flat_settings=True, ) print(resp)
response = client.indices.get_settings( index: 'my-index-000001', name: 'index.routing.allocation.include._tier_preference', flat_settings: true ) puts response
const response = await client.indices.getSettings({ index: "my-index-000001", name: "index.routing.allocation.include._tier_preference", flat_settings: "true", }); console.log(response);
GET /my-index-000001/_settings/index.routing.allocation.include._tier_preference?flat_settings
The response will look like this:
{ "my-index-000001": { "settings": { "index.routing.allocation.include._tier_preference": "data_warm,data_hot" } } }
Represents a comma separated list of data tier node roles this index is allowed
to be allocated on, the first one in the list being the one with the higher priority
i.e. the tier the index is targeting.
e.g. in this example the tier preference is |
Alternatively, if adding more nodes to the Elasticsearch cluster is not desired,
inspecting the system-wide cluster.routing.allocation.total_shards_per_node
cluster setting and increasing the configured value:
-
Inspect the
cluster.routing.allocation.total_shards_per_node
cluster setting for the index with unassigned shards:resp = client.cluster.get_settings( flat_settings=True, ) print(resp)
response = client.cluster.get_settings( flat_settings: true ) puts response
const response = await client.cluster.getSettings({ flat_settings: "true", }); console.log(response);
GET /_cluster/settings?flat_settings
The response will look like this:
-
Increase the value for the total number of shards that can be assigned on one node to a higher value:
resp = client.cluster.put_settings( persistent={ "cluster.routing.allocation.total_shards_per_node": 400 }, ) print(resp)
response = client.cluster.put_settings( body: { persistent: { 'cluster.routing.allocation.total_shards_per_node' => 400 } } ) puts response
const response = await client.cluster.putSettings({ persistent: { "cluster.routing.allocation.total_shards_per_node": 400, }, }); console.log(response);