High JVM memory pressure
editHigh JVM memory pressure
editHigh JVM memory usage can degrade cluster performance and trigger circuit breaker errors. To prevent this, we recommend taking steps to reduce memory pressure if a node’s JVM memory usage consistently exceeds 85%.
Diagnose high JVM memory pressure
editCheck JVM memory pressure
From your deployment menu, click Elasticsearch. Under Instances, each instance displays a JVM memory pressure indicator. When the JVM memory pressure reaches 75%, the indicator turns red.
You can also use the nodes stats API to calculate the current JVM memory pressure for each node.
resp = client.nodes.stats( filter_path="nodes.*.jvm.mem.pools.old", ) print(resp)
response = client.nodes.stats( filter_path: 'nodes.*.jvm.mem.pools.old' ) puts response
const response = await client.nodes.stats({ filter_path: "nodes.*.jvm.mem.pools.old", }); console.log(response);
GET _nodes/stats?filter_path=nodes.*.jvm.mem.pools.old
Use the response to calculate memory pressure as follows:
JVM Memory Pressure = used_in_bytes
/ max_in_bytes
To calculate the current JVM memory pressure for each node, use the nodes stats API.
resp = client.nodes.stats( filter_path="nodes.*.jvm.mem.pools.old", ) print(resp)
response = client.nodes.stats( filter_path: 'nodes.*.jvm.mem.pools.old' ) puts response
const response = await client.nodes.stats({ filter_path: "nodes.*.jvm.mem.pools.old", }); console.log(response);
GET _nodes/stats?filter_path=nodes.*.jvm.mem.pools.old
Use the response to calculate memory pressure as follows:
JVM Memory Pressure = used_in_bytes
/ max_in_bytes
Check garbage collection logs
As memory usage increases, garbage collection becomes more frequent and takes
longer. You can track the frequency and length of garbage collection events in
elasticsearch.log
. For example, the following event states Elasticsearch
spent more than 50% (21 seconds) of the last 40 seconds performing garbage
collection.
[timestamp_short_interval_from_last][INFO ][o.e.m.j.JvmGcMonitorService] [node_id] [gc][number] overhead, spent [21s] collecting in the last [40s]
Capture a JVM heap dump
To determine the exact reason for the high JVM memory pressure, capture a heap dump of the JVM while its memory usage is high, and also capture the garbage collector logs covering the same time period.
Reduce JVM memory pressure
editThis section contains some common suggestions for reducing JVM memory pressure.
Reduce your shard count
Every shard uses memory. In most cases, a small set of large shards uses fewer resources than many small shards. For tips on reducing your shard count, see Size your shards.
Expensive searches can use large amounts of memory. To better track expensive searches on your cluster, enable slow logs.
Expensive searches may have a large size
argument,
use aggregations with a large number of buckets, or include
expensive queries. To prevent expensive
searches, consider the following setting changes:
-
Lower the
size
limit using theindex.max_result_window
index setting. - Decrease the maximum number of allowed aggregation buckets using the search.max_buckets cluster setting.
-
Disable expensive queries using the
search.allow_expensive_queries
cluster setting. -
Set a default search timeout using the
search.default_search_timeout
cluster setting.
resp = client.indices.put_settings( settings={ "index.max_result_window": 5000 }, ) print(resp) resp1 = client.cluster.put_settings( persistent={ "search.max_buckets": 20000, "search.allow_expensive_queries": False }, ) print(resp1)
response = client.indices.put_settings( body: { 'index.max_result_window' => 5000 } ) puts response response = client.cluster.put_settings( body: { persistent: { 'search.max_buckets' => 20_000, 'search.allow_expensive_queries' => false } } ) puts response
const response = await client.indices.putSettings({ settings: { "index.max_result_window": 5000, }, }); console.log(response); const response1 = await client.cluster.putSettings({ persistent: { "search.max_buckets": 20000, "search.allow_expensive_queries": false, }, }); console.log(response1);
PUT _settings { "index.max_result_window": 5000 } PUT _cluster/settings { "persistent": { "search.max_buckets": 20000, "search.allow_expensive_queries": false } }
Prevent mapping explosions
Defining too many fields or nesting fields too deeply can lead to mapping explosions that use large amounts of memory. To prevent mapping explosions, use the mapping limit settings to limit the number of field mappings.
Spread out bulk requests
While more efficient than individual requests, large bulk indexing or multi-search requests can still create high JVM memory pressure. If possible, submit smaller requests and allow more time between them.
Upgrade node memory
Heavy indexing and search loads can cause high JVM memory pressure. To better handle heavy workloads, upgrade your nodes to increase their memory capacity.