Total number of shards per node has been reached

edit

Total number of shards per node has been reached

edit

Elasticsearch tries to take advantage of all the available resources by distributing data (index shards) amongst the cluster nodes.

Users might want to influence this data distribution by configuring the cluster.routing.allocation.total_shards_per_node system setting to restrict the number of shards that can be hosted on a single node in the system, regardless of the index. Various configurations limiting how many shards can be hosted on a single node can lead to shards being unassigned due to the cluster not having enough nodes to satisfy the configuration.

In order to fix this follow the next steps:

In order to get the shards assigned we’ll need to increase the number of shards that can be collocated on a node in the cluster. We’ll achieve this by inspecting the system-wide cluster.routing.allocation.total_shards_per_node cluster setting and increasing the configured value.

Use Kibana

  1. Log in to the Elastic Cloud console.
  2. On the Elasticsearch Service panel, click the name of your deployment.

    If the name of your deployment is disabled your Kibana instances might be unhealthy, in which case please contact Elastic Support. If your deployment doesn’t include Kibana, all you need to do is enable it first.

  3. Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to Dev Tools > Console.

    Kibana Console
  4. Inspect the cluster.routing.allocation.total_shards_per_node cluster setting:

    resp = client.cluster.get_settings(
        flat_settings=True,
    )
    print(resp)
    response = client.cluster.get_settings(
      flat_settings: true
    )
    puts response
    const response = await client.cluster.getSettings({
      flat_settings: "true",
    });
    console.log(response);
    GET /_cluster/settings?flat_settings

    The response will look like this:

    {
      "persistent": {
        "cluster.routing.allocation.total_shards_per_node": "300" 
      },
      "transient": {}
    }

    Represents the current configured value for the total number of shards that can reside on one node in the system.

  5. Increase the value for the total number of shards that can be assigned on one node to a higher value:

    resp = client.cluster.put_settings(
        persistent={
            "cluster.routing.allocation.total_shards_per_node": 400
        },
    )
    print(resp)
    response = client.cluster.put_settings(
      body: {
        persistent: {
          'cluster.routing.allocation.total_shards_per_node' => 400
        }
      }
    )
    puts response
    const response = await client.cluster.putSettings({
      persistent: {
        "cluster.routing.allocation.total_shards_per_node": 400,
      },
    });
    console.log(response);
    PUT _cluster/settings
    {
      "persistent" : {
        "cluster.routing.allocation.total_shards_per_node" : 400 
      }
    }

    The new value for the system-wide total_shards_per_node configuration is increased from the previous value of 300 to 400. The total_shards_per_node configuration can also be set to null, which represents no upper bound with regards to how many shards can be collocated on one node in the system.