Diagnose unassigned shards
editDiagnose unassigned shards
editThere are multiple reasons why shards might get unassigned, ranging from misconfigured allocation settings to lack of disk space.
In order to diagnose the unassigned shards in your deployment use the following steps:
In order to diagnose the unassigned shards, follow the next steps:
Use Kibana
- Log in to the Elastic Cloud console.
-
On the Elasticsearch Service panel, click the name of your deployment.
If the name of your deployment is disabled your Kibana instances might be unhealthy, in which case please contact Elastic Support. If your deployment doesn’t include Kibana, all you need to do is enable it first.
-
Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to Dev Tools > Console.
-
View the unassigned shards using the cat shards API.
resp = client.cat.shards( v=True, h="index,shard,prirep,state,node,unassigned.reason", s="state", ) print(resp)
response = client.cat.shards( v: true, h: 'index,shard,prirep,state,node,unassigned.reason', s: 'state' ) puts response
const response = await client.cat.shards({ v: "true", h: "index,shard,prirep,state,node,unassigned.reason", s: "state", }); console.log(response);
GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state
The response will look like this:
[ { "index": "my-index-000001", "shard": "0", "prirep": "p", "state": "UNASSIGNED", "node": null, "unassigned.reason": "INDEX_CREATED" } ]
Unassigned shards have a
state
ofUNASSIGNED
. Theprirep
value isp
for primary shards andr
for replicas.The index in the example has a primary shard unassigned.
-
To understand why an unassigned shard is not being assigned and what action you must take to allow Elasticsearch to assign it, use the cluster allocation explanation API.
resp = client.cluster.allocation_explain( index="my-index-000001", shard=0, primary=True, ) print(resp)
response = client.cluster.allocation_explain( body: { index: 'my-index-000001', shard: 0, primary: true } ) puts response
const response = await client.cluster.allocationExplain({ index: "my-index-000001", shard: 0, primary: true, }); console.log(response);
The index we want to diagnose.
The unassigned shard ID.
Indicates that we are diagnosing a primary shard.
The response will look like this:
{ "index" : "my-index-000001", "shard" : 0, "primary" : true, "current_state" : "unassigned", "unassigned_info" : { "reason" : "INDEX_CREATED", "at" : "2022-01-04T18:08:16.600Z", "last_allocation_status" : "no" }, "can_allocate" : "no", "allocate_explanation" : "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.", "node_allocation_decisions" : [ { "node_id" : "8qt2rY-pT6KNZB3-hGfLnw", "node_name" : "node-0", "transport_address" : "127.0.0.1:9401", "roles": ["data_content", "data_hot"], "node_attributes" : {}, "node_decision" : "no", "weight_ranking" : 1, "deciders" : [ { "decider" : "filter", "decision" : "NO", "explanation" : "node does not match index setting [index.routing.allocation.include] filters [_name:\"nonexistent_node\"]" } ] } ] }
The current state of the shard.
The reason for the shard originally becoming unassigned.
Whether to allocate the shard.
Whether to allocate the shard to the particular node.
The decider which led to the
no
decision for the node.An explanation as to why the decider returned a
no
decision, with a helpful hint pointing to the setting that led to the decision. -
The explanation in our case indicates the index allocation configurations are not correct. To review your allocation settings, use the get index settings and cluster get settings APIs.
resp = client.indices.get_settings( index="my-index-000001", flat_settings=True, include_defaults=True, ) print(resp) resp1 = client.cluster.get_settings( flat_settings=True, include_defaults=True, ) print(resp1)
response = client.indices.get_settings( index: 'my-index-000001', flat_settings: true, include_defaults: true ) puts response response = client.cluster.get_settings( flat_settings: true, include_defaults: true ) puts response
const response = await client.indices.getSettings({ index: "my-index-000001", flat_settings: "true", include_defaults: "true", }); console.log(response); const response1 = await client.cluster.getSettings({ flat_settings: "true", include_defaults: "true", }); console.log(response1);
GET my-index-000001/_settings?flat_settings=true&include_defaults=true GET _cluster/settings?flat_settings=true&include_defaults=true
- Change the settings using the update index settings and cluster update settings APIs to the correct values in order to allow the index to be allocated.
For more guidance on fixing the most common causes for unassinged shards please follow this guide or contact Elastic Support.
In order to diagnose the unassigned shards follow the next steps:
-
View the unassigned shards using the cat shards API.
resp = client.cat.shards( v=True, h="index,shard,prirep,state,node,unassigned.reason", s="state", ) print(resp)
response = client.cat.shards( v: true, h: 'index,shard,prirep,state,node,unassigned.reason', s: 'state' ) puts response
const response = await client.cat.shards({ v: "true", h: "index,shard,prirep,state,node,unassigned.reason", s: "state", }); console.log(response);
GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state
The response will look like this:
[ { "index": "my-index-000001", "shard": "0", "prirep": "p", "state": "UNASSIGNED", "node": null, "unassigned.reason": "INDEX_CREATED" } ]
Unassigned shards have a
state
ofUNASSIGNED
. Theprirep
value isp
for primary shards andr
for replicas.The index in the example has a primary shard unassigned.
-
To understand why an unassigned shard is not being assigned and what action you must take to allow Elasticsearch to assign it, use the cluster allocation explanation API.
resp = client.cluster.allocation_explain( index="my-index-000001", shard=0, primary=True, ) print(resp)
response = client.cluster.allocation_explain( body: { index: 'my-index-000001', shard: 0, primary: true } ) puts response
const response = await client.cluster.allocationExplain({ index: "my-index-000001", shard: 0, primary: true, }); console.log(response);
The index we want to diagnose.
The unassigned shard ID.
Indicates that we are diagnosing a primary shard.
The response will look like this:
{ "index" : "my-index-000001", "shard" : 0, "primary" : true, "current_state" : "unassigned", "unassigned_info" : { "reason" : "INDEX_CREATED", "at" : "2022-01-04T18:08:16.600Z", "last_allocation_status" : "no" }, "can_allocate" : "no", "allocate_explanation" : "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.", "node_allocation_decisions" : [ { "node_id" : "8qt2rY-pT6KNZB3-hGfLnw", "node_name" : "node-0", "transport_address" : "127.0.0.1:9401", "roles": ["data_content", "data_hot"] "node_attributes" : {}, "node_decision" : "no", "weight_ranking" : 1, "deciders" : [ { "decider" : "filter", "decision" : "NO", "explanation" : "node does not match index setting [index.routing.allocation.include] filters [_name:\"nonexistent_node\"]" } ] } ] }
The current state of the shard.
The reason for the shard originally becoming unassigned.
Whether to allocate the shard.
Whether to allocate the shard to the particular node.
The decider which led to the
no
decision for the node.An explanation as to why the decider returned a
no
decision, with a helpful hint pointing to the setting that led to the decision. -
The explanation in our case indicates the index allocation configurations are not correct. To review your allocation settings, use the get index settings and cluster get settings APIs.
resp = client.indices.get_settings( index="my-index-000001", flat_settings=True, include_defaults=True, ) print(resp) resp1 = client.cluster.get_settings( flat_settings=True, include_defaults=True, ) print(resp1)
response = client.indices.get_settings( index: 'my-index-000001', flat_settings: true, include_defaults: true ) puts response response = client.cluster.get_settings( flat_settings: true, include_defaults: true ) puts response
const response = await client.indices.getSettings({ index: "my-index-000001", flat_settings: "true", include_defaults: "true", }); console.log(response); const response1 = await client.cluster.getSettings({ flat_settings: "true", include_defaults: "true", }); console.log(response1);
GET my-index-000001/_settings?flat_settings=true&include_defaults=true GET _cluster/settings?flat_settings=true&include_defaults=true
- Change the settings using the update index settings and cluster update settings APIs to the correct values in order to allow the index to be allocated.
For more guidance on fixing the most common causes for unassinged shards please follow this guide.