IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Hot spotting Add a missing tier to the system »
Elastic Docs ›Elasticsearch Guide [8.8] ›Troubleshooting

Diagnose unassigned shards

edit

Diagnose unassigned shards

edit

There are multiple reasons why shards might get unassigned, ranging from misconfigured allocation settings to lack of disk space.

In order to diagnose the unassigned shards in your deployment use the following steps:

In order to diagnose the unassigned shards, follow the next steps:

Use Kibana

  1. Log in to the Elastic Cloud console.

  2. On the Elasticsearch Service panel, click the name of your deployment.

    If the name of your deployment is disabled your Kibana instances might be unhealthy, in which case please contact Elastic Support. If your deployment doesn’t include Kibana, all you need to do is enable it first.

  3. Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to Dev Tools > Console.

    Kibana Console

  4. View the unassigned shards using the cat shards API.

    response = client.cat.shards(
  v: true,
  h: 'index,shard,prirep,state,node,unassigned.reason',
  s: 'state'
)
puts response
    GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state

    The response will look like this:

    [
  {
    "index": "my-index-000001",
    "shard": "0",
    "prirep": "p",
    "state": "UNASSIGNED",
    "node": null,
    "unassigned.reason": "INDEX_CREATED"
  }
]

    Unassigned shards have a state of UNASSIGNED. The prirep value is p for primary shards and r for replicas.

    The index in the example has a primary shard unassigned.

  5. To understand why an unassigned shard is not being assigned and what action you must take to allow Elasticsearch to assign it, use the cluster allocation explanation API.

    response = client.cluster.allocation_explain(
  body: {
    index: 'my-index-000001',
    shard: 0,
    primary: true
  }
)
puts response
    GET _cluster/allocation/explain
{
  "index": "my-index-000001", 
  "shard": 0, 
  "primary": true 
}

    The index we want to diagnose.

    The unassigned shard ID.

    Indicates that we are diagnosing a primary shard.

    The response will look like this:

    {
  "index" : "my-index-000001",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",                 
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",                   
    "at" : "2022-01-04T18:08:16.600Z",
    "last_allocation_status" : "no"
  },
  "can_allocate" : "no",                          
  "allocate_explanation" : "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.",
  "node_allocation_decisions" : [
    {
      "node_id" : "8qt2rY-pT6KNZB3-hGfLnw",
      "node_name" : "node-0",
      "transport_address" : "127.0.0.1:9401",
      "node_attributes" : {},
      "node_decision" : "no",                     
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "filter",                   
          "decision" : "NO",
          "explanation" : "node does not match index setting [index.routing.allocation.include] filters [_name:\"nonexistent_node\"]"  
        }
      ]
    }
  ]
}

    The current state of the shard.

    The reason for the shard originally becoming unassigned.

    Whether to allocate the shard.

    Whether to allocate the shard to the particular node.

    The decider which led to the no decision for the node.

    An explanation as to why the decider returned a no decision, with a helpful hint pointing to the setting that led to the decision.

  6. The explanation in our case indicates the index allocation configurations are not correct. To review your allocation settings, use the get index settings and cluster get settings APIs.

    response = client.indices.get_settings(
  index: 'my-index-000001',
  flat_settings: true,
  include_defaults: true
)
puts response

response = client.indices.get_settings(
  flat_settings: true,
  include_defaults: true
)
puts response
    GET my-index-000001/_settings?flat_settings=true&include_defaults=true

GET _cluster/settings?flat_settings=true&include_defaults=true
  7. Change the settings using the update index settings and cluster update settings APIs to the correct values in order to allow the index to be allocated.

For more guidance on fixing the most common causes for unassinged shards please follow this guide or contact Elastic Support.

« Hot spotting Add a missing tier to the system »