Elasticsearch autoscaling

ECK can leverage the autoscaling API introduced in Elasticsearch 7.11 to adjust automatically the number of Pods and the allocated resources in a tier. Currently, autoscaling is supported for Elasticsearch data tiers and machine learning nodes.

To enable autoscaling on an Elasticsearch cluster, you need to define one or more autoscaling policies. Each autoscaling policy applies to one or more NodeSets which share the same set of roles specified in the node.roles setting in the Elasticsearch configuration.

Autoscaling policies can be defined in the elasticsearch.alpha.elastic.co/autoscaling-spec annotation of the Elasticsearch resource. Each autoscaling policy must have the following fields:

{
    "policies": [{
      "name": "data-ingest",
      "roles": ["data", "ingest" , "transform"],
      "resources": {
          "nodeCount": { "min": 3, "max": 8 },
          "cpu": { "min": 2, "max": 8 },
          "memory": { "min": "2Gi", "max": "16Gi" },
          "storage": { "min": "64Gi", "max": "512Gi" }
      }
    },
    {
      "name": "ml",
      "roles": ["ml"],
      "resources": {
          "nodeCount": { "min": 1, "max": 9 },
          "cpu": { "min": 1, "max": 4 },
          "memory": { "min": "2Gi", "max": "8Gi" }
      }
    }]
}

In the case of storage the following restrictions apply:

The value set for memory and CPU limits are computed by applying a ratio to the calculated resource request. For memory, the default ratio between the request and the limit is 1. This means that request and limit have the same value. For CPU the default ratio is 0, which means that no limit is set. You can change the default ratio between the request and the limit for both the CPU and memory ranges by using the requestsToLimitsRatio field.

For example, you can remove the memory limit and set a CPU limit to twice the value of the request, as follows:

{
    "policies": [{
        "name": "data-ingest-hot",
        "roles": ["data_hot", "ingest", "transform"],
        "resources": {
            "nodeCount": {
                "min": 2,
                "max": 5
            },
            "cpu": {
                "min": 1,
                "max": 2,
                "requestsToLimitsRatio": 2
            },
            "memory": {
                "min": "2Gi",
                "max": "6Gi",
                "requestsToLimitsRatio": 0
            }
        }
    }]
}

You can find a complete example in the ECK GitHub repository which will also show you how to fine-tune the autoscaling deciders.

The Elasticsearch autoscaling capacity endpoint is polled every minute by the operator. This interval duration can be controlled using the pollingPeriod field in the autoscaling specification:

{
    "pollingPeriod": "42s",
    "policies": [{
        "name": "data-ingest-hot",
        "roles": ["data_hot", "ingest", "transform"],
        "resources": {
            "nodeCount": {
                "min": 2,
                "max": 5
            },
            "cpu": {
                "min": 1,
                "max": 2
            },
            "memory": {
                "min": "2Gi",
                "max": "6Gi"
            }
        }
    }]
}

In addition to the logs generated by the operator, an autoscaling status is stored in the elasticsearch.alpha.elastic.co/autoscaling-status annotation. The autoscaling status is a JSON document which describes the expected resources for each NodeSet managed by an autoscaling policy. It may also contain important messages about the state of the tier.

{
	"policies": [
		{
			"name": "data-ingest-hot",
			"nodeSets": [{
				"name": "data-ingest-hot",
				"nodeCount": 5
			}],
			"resources": {
				"limits": {
					"cpu": "2",
					"memory": "6Gi"
				},
				"requests": {
					"cpu": "2",
					"memory": "6Gi",
					"storage": "6Gi"
				}
			},
			"state": [{
				"type": "HorizontalScalingLimitReached",
				"messages": [
					"Can't provide total required storage 32588740338, max number of nodes is 5, requires 6 nodes"
				]
			}],
			"lastModificationTime": "2021-03-09T17:01:25Z"
		}
	]
}

Important events are also reported through Kubernetes events, for example when the maximum autoscaling size limit is reached:

> kubectl get events

40m  Warning  HorizontalScalingLimitReached  elasticsearch/sample   Can't provide total required storage 32588740338, max number of nodes is 5, requires 6 nodes

You can disable autoscaling at any time by removing the elasticsearch.alpha.elastic.co/autoscaling-spec annotation from the Elasticsearch resource metadata.

For machine learning the following settings are not automatically reset:

You should adjust those settings manually to match the size of your deployment when you disable autoscaling.