Increase the disk capacity of data nodes

If autoscaling is available but not enabled, please enable it. You can do this by clicking the button Enable autoscaling on a banner like the one below:

Or you can go to Actions > Edit deployment, check the checkbox Autoscale and click save at the bottom of the page.

If autoscaling has succeeded the cluster should return to healthy status. If the cluster is still out of disk, please check if autoscaling has reached its limits. You will be notified about this by the following banner:

or you can go to Actions > Edit deployment and look for the label LIMIT REACHED as shown below:

If you are seeing the banner click Update autoscaling settings to go to the Edit page. Otherwise, you are already in the Edit page, click Edit settings to increase the autoscaling limits. After you perform the change click save at the bottom of the page.

First, retrieve the relevant disk thresholds that will indicate how much space should be available. The relevant thresholds are the high watermark for all the tiers apart from the frozen one and the frozen flood stage watermark for the frozen tier. The following example demonstrates disk shortage in the hot tier, so we will only retrieve the high watermark:

resp = client.cluster.get_settings(
    include_defaults=True,
    filter_path="*.cluster.routing.allocation.disk.watermark.high*",
)
print(resp)

response = client.cluster.get_settings(
  include_defaults: true,
  filter_path: '*.cluster.routing.allocation.disk.watermark.high*'
)
puts response

const response = await client.cluster.getSettings({
  include_defaults: "true",
  filter_path: "*.cluster.routing.allocation.disk.watermark.high*",
});
console.log(response);

GET _cluster/settings?include_defaults&filter_path=*.cluster.routing.allocation.disk.watermark.high*

The response will look like this:

{
  "defaults": {
    "cluster": {
      "routing": {
        "allocation": {
          "disk": {
            "watermark": {
              "high": "90%",
              "high.max_headroom": "150GB"
            }
          }
        }
      }
    }
  }
}

The above means that in order to resolve the disk shortage we need to either drop our disk usage below the 90% or have more than 150GB available, read more on how this threshold works here.

The next step is to find out the current disk usage, this will indicate how much extra space is needed. For simplicity, our example has one node, but you can apply the same for every node over the relevant threshold.

resp = client.cat.allocation(
    v=True,
    s="disk.avail",
    h="node,disk.percent,disk.avail,disk.total,disk.used,disk.indices,shards",
)
print(resp)

response = client.cat.allocation(
  v: true,
  s: 'disk.avail',
  h: 'node,disk.percent,disk.avail,disk.total,disk.used,disk.indices,shards'
)
puts response

const response = await client.cat.allocation({
  v: "true",
  s: "disk.avail",
  h: "node,disk.percent,disk.avail,disk.total,disk.used,disk.indices,shards",
});
console.log(response);

GET _cat/allocation?v&s=disk.avail&h=node,disk.percent,disk.avail,disk.total,disk.used,disk.indices,shards

The response will look like this:

node                disk.percent disk.avail disk.total disk.used disk.indices shards
instance-0000000000           91     4.6gb       35gb    31.1gb       29.9gb    111

The high watermark configuration indicates that the disk usage needs to drop below 90%. To achieve this, 2 things are possible:

In the case of adding another data node, the cluster will not recover immediately. It might take some time to relocate some shards to the new node. You can check the progress here:

resp = client.cat.shards(
    v=True,
    h="state,node",
    s="state",
)
print(resp)

response = client.cat.shards(
  v: true,
  h: 'state,node',
  s: 'state'
)
puts response

const response = await client.cat.shards({
  v: "true",
  h: "state,node",
  s: "state",
});
console.log(response);

GET /_cat/shards?v&h=state,node&s=state

If in the response the shards' state is RELOCATING, it means that shards are still moving. Wait until all shards turn to STARTED or until the health disk indicator turns to green.