Managing and troubleshooting Elasticsearch memory

With Elastic Cloud delivering solutions like Observability, Security, and Search, we've broadened the users who use Elastic Cloud beyond full ops teams to include data engineers, security teams, and consultants. As an Elastic support representative, I’ve enjoyed engaging with a diverse range of users and use cases.

With a wider audience, I’m seeing more questions about managing resource allocation, in particular troubleshooting allocation health and avoiding circuit breakers. I get it! When I started with Elasticsearch, I had the same questions. It was my first intro to managing Java heap and time series database shards and scaling my own infrastructure.

When I joined Elastic, I loved that on top of documentation, we had blogs and tutorials so I could onboard quickly. But then I struggled my first month to correlate my theoretical knowledge to the errors users would send through my ticket queue. Eventually I figured out, like other support reps, that a lot of the reported errors were just symptoms of allocation issues and the same seven-ish links would bring users up to speed to successfully manage their resource allocation.

Speaking as a support rep, I’m going to go over the top allocation management theory links we send users, the top symptoms we see, and where we direct users to update their configurations to resolve their resource allocation issues.

2. As an Elasticsearch environment variable in your docker-compose:

3. Via our Elastic Cloud Hosted > Deployment > Edit view. Note: The drop down assigns physical memory and roughly half will be allotted to the heap.

If you’re currently experiencing performance issues with your cluster, it will most likely come down to the usual suspects:

• Configuration issues: Undersized master nodes, no ILM policy
• Volume induced: High request pace/load, overlapping expensive queries/writes

And to: _cluster/health.

If any shards report >0 outside active_shards or active_primary_shards, you’ve pinpointed a cause for performance issues.

Most commonly if this reports an issue, it will be unassigned_shards>0. If these shards are primary, your cluster will report as status:red, and if only replicas, it will report as status:yellow. (This is why setting replicas on indices is important — if the cluster encounters an issue, it can recover rather than experience data loss.)Let’s pretend we have a status:yellow with a single unassigned shard. To investigate, we’d take a look at which index shard is having trouble via _cat/shards.

So this will be for our non-system index logs, which have an unassigned replica shard. Let’s see what’s giving it grief by running _cluster/allocation/explain. (Pro tip: When you escalate to support, this is exactly what we do.)

GET _cluster/allocation/explain?pretty&filter_path=index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.*

This error message points to data_hot, which is part of an index lifecycle management (ILM) policy and indicates that our ILM policy is incongruent with our current index settings. In this case, the cause of this error is from setting up a hot-warm ILM policy without having designated hot-warm nodes. (I needed to guarantee something would fail, so this is me forcing error examples for y’all. For more information, see this example troubleshooting video for resolution walkthrough.)

If you run this command when you don’t have any unassigned shards, you’ll get a 400 error saying unable to find any unassigned shards to explain because nothing’s wrong to report on.If you get a non-logic cause (e.g., a temporary network error like node left cluster during allocation), then you can use Elastic’s handy-dandy _cluster/reroute.

This request without customizations starts an asynchronous background process that attempts to allocate all current state:UNASSIGNED shards. (Don’t be like me and not wait for it to finish before you contact dev, because I thought it would be instantaneous and coincidentally escalate just in time for them to say nothing’s wrong because nothing was anymore.) For more information, see this troubleshooting video for monitoring Allocation Health.

To investigate, take a look at your heap.percent, either by looking at _cat/nodes:

Or if you’ve previously enabled it, navigate to Kibana > Stack Monitoring.

If you've confirmed you're hitting your memory circuit breakers, you'll want to consider increasing heap temporarily to give yourself breathing room to investigate. When investigating root cause, look through your audit logging, slow logging, clusterlogs, or elasticsearch.log for the preceding consecutive events. You'll be looking for:

• Expensive queries, especially:
  • High bucket aggregations
    • I felt so silly when I found out that searches temporarily allocate a certain portion of your heap before they run the query based on the search size or bucket dimensions, so setting 10,000,000 really was giving my ops team heartburn.
  • non-optimized mappings
    • The second reason to feel silly was when I thought doing hierarchical reporting would search better than flattened out data (it does not).
• Request volume/pace: Usually batch or async queries

Or you can calculate it from _nodes/stats:

Where:

A potential symptom of this is high frequency and long duration from garbage collector (gc) events in your elasticsearch.log:

If you confirm this scenario, you’ll need to take a look either at scaling your cluster or at reducing the demands hitting it. You’ll want to investigate/consider:

• Increasing heap resources (heap/node; number of nodes)
• Decreasing shards (delete unnecessary/old data; use ILM to put data into warm/cold storage so you can shrink it; turn off replicas for data you don’t care if you lose)

Originally published April 27, 2021; updated November 5, 2024.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.