High JVM memory pressure

High JVM memory usage can degrade cluster performance and trigger circuit breaker errors. To prevent this, we recommend taking steps to reduce memory pressure if a node’s JVM memory usage consistently exceeds 85%.

Check JVM memory pressure

response = client.nodes.stats(
  filter_path: 'nodes.*.jvm.mem.pools.old'
)
puts response

GET _nodes/stats?filter_path=nodes.*.jvm.mem.pools.old

response = client.nodes.stats(
  filter_path: 'nodes.*.jvm.mem.pools.old'
)
puts response

GET _nodes/stats?filter_path=nodes.*.jvm.mem.pools.old

Check garbage collection logs

As memory usage increases, garbage collection becomes more frequent and takes longer. You can track the frequency and length of garbage collection events in elasticsearch.log. For example, the following event states Elasticsearch spent more than 50% (21 seconds) of the last 40 seconds performing garbage collection.

[timestamp_short_interval_from_last][INFO ][o.e.m.j.JvmGcMonitorService] [node_id] [gc][number] overhead, spent [21s] collecting in the last [40s]

Capture a JVM heap dump

To determine the exact reason for the high JVM memory pressure, capture a heap dump of the JVM while its memory usage is high.

This section contains some common suggestions for reducing JVM memory pressure.

Reduce your shard count

Every shard uses memory. In most cases, a small set of large shards uses fewer resources than many small shards. For tips on reducing your shard count, see Size your shards.

Avoid expensive searches

Expensive searches can use large amounts of memory. To better track expensive searches on your cluster, enable slow logs.

Expensive searches may have a large size argument, use aggregations with a large number of buckets, or include expensive queries. To prevent expensive searches, consider the following setting changes:

PUT _settings
{
  "index.max_result_window": 5000
}

PUT _cluster/settings
{
  "persistent": {
    "search.max_buckets": 20000,
    "search.allow_expensive_queries": false
  }
}

Prevent mapping explosions

Defining too many fields or nesting fields too deeply can lead to mapping explosions that use large amounts of memory. To prevent mapping explosions, use the mapping limit settings to limit the number of field mappings.

Spread out bulk requests

While more efficient than individual requests, large bulk indexing or multi-search requests can still create high JVM memory pressure. If possible, submit smaller requests and allow more time between them.

Upgrade node memory

Heavy indexing and search loads can cause high JVM memory pressure. To better handle heavy workloads, upgrade your nodes to increase their memory capacity.