Red or yellow cluster health status

A red or yellow cluster health status indicates one or more shards are not assigned to a node.

When your cluster has a red or yellow health status, it will continue to process searches and indexing where possible, but may delay certain management and cleanup activities until the cluster returns to green health status. For instance, some ILM actions require the index on which they operate to have a green health status.

In many cases, your cluster will recover to green health status automatically. If the cluster doesn’t automatically recover, then you must manually address the remaining problems so management and cleanup activities can proceed.

Check your cluster status

Use the cluster health API.

response = client.cluster.health(
  filter_path: 'status,*_shards'
)
puts response

GET _cluster/health?filter_path=status,*_shards

A healthy cluster has a green status and zero unassigned_shards. A yellow status means only replicas are unassigned. A red status means one or more primary shards are unassigned.

View unassigned shards

To view unassigned shards, use the cat shards API.

response = client.cat.shards(
  v: true,
  h: 'index,shard,prirep,state,node,unassigned.reason',
  s: 'state'
)
puts response

GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state

Unassigned shards have a state of UNASSIGNED. The prirep value is p for primary shards and r for replicas.

To understand why an unassigned shard is not being assigned and what action you must take to allow Elasticsearch to assign it, use the cluster allocation explanation API.

response = client.cluster.allocation_explain(
  filter_path: 'index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.*',
  body: {
    index: 'my-index',
    shard: 0,
    primary: false
  }
)
puts response

GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.*
{
  "index": "my-index",
  "shard": 0,
  "primary": false
}

A shard can become unassigned for several reasons. The following tips outline the most common causes and their solutions.

You typically disable allocation during a restart or other cluster maintenance. If you forgot to re-enable allocation afterward, Elasticsearch will be unable to assign shards. To re-enable allocation, reset the cluster.routing.allocation.enable cluster setting.

response = client.cluster.put_settings(
  body: {
    persistent: {
      'cluster.routing.allocation.enable' => nil
    }
  }
)
puts response

PUT _cluster/settings
{
  "persistent" : {
    "cluster.routing.allocation.enable" : null
  }
}

Shards often become unassigned when a data node leaves the cluster. This can occur for several reasons, ranging from connectivity issues to hardware failure. After you resolve the issue and recover the node, it will rejoin the cluster. Elasticsearch will then automatically allocate any unassigned shards.

To avoid wasting resources on temporary issues, Elasticsearch delays allocation by one minute by default. If you’ve recovered a node and don’t want to wait for the delay period, you can call the cluster reroute API with no arguments to start the allocation process. The process runs asynchronously in the background.

response = client.cluster.reroute(
  metric: 'none'
)
puts response

POST _cluster/reroute?metric=none

Misconfigured allocation settings can result in an unassigned primary shard. These settings include:

To review your allocation settings, use the get index settings and cluster get settings APIs.

response = client.indices.get_settings(
  index: 'my-index',
  flat_settings: true,
  include_defaults: true
)
puts response

response = client.cluster.get_settings(
  flat_settings: true,
  include_defaults: true
)
puts response

GET my-index/_settings?flat_settings=true&include_defaults=true

GET _cluster/settings?flat_settings=true&include_defaults=true

You can change the settings using the update index settings and cluster update settings APIs.

To protect against hardware failure, Elasticsearch will not assign a replica to the same node as its primary shard. If no other data nodes are available to host the replica, it remains unassigned. To fix this, you can:

response = client.indices.put_settings(
  body: {
    'index.number_of_replicas' => 1
  }
)
puts response

PUT _settings
{
  "index.number_of_replicas": 1
}

Elasticsearch uses a low disk watermark to ensure data nodes have enough disk space for incoming shards. By default, Elasticsearch does not allocate shards to nodes using more than 85% of disk space.

To check the current disk space of your nodes, use the cat allocation API.

response = client.cat.allocation(
  v: true,
  h: 'node,shards,disk.*'
)
puts response

GET _cat/allocation?v=true&h=node,shards,disk.*

If your nodes are running low on disk space, you have a few options:

Shard allocation requires JVM heap memory. High JVM memory pressure can trigger circuit breakers that stop allocation and leave shards unassigned. See High JVM memory pressure.

If a node containing a primary shard is lost, Elasticsearch can typically replace it using a replica on another node. If you can’t recover the node and replicas don’t exist or are irrecoverable, you’ll need to re-add the missing data from a snapshot or the original data source.

Use the cluster reroute API to manually allocate the unassigned primary shard to another data node in the same tier. Set accept_data_loss to true.

POST _cluster/reroute?metric=none
{
  "commands": [
    {
      "allocate_empty_primary": {
        "index": "my-index",
        "shard": 0,
        "node": "my-node",
        "accept_data_loss": "true"
      }
    }
  ]
}

If you backed up the missing index data to a snapshot, use the restore snapshot API to restore the individual index. Alternatively, you can index the missing data from the original data source.