JVM heap dumps

Ensure sufficient storage

Elasticsearch is configured by default to take heap dumps on out-of-memory exceptions to the default data directory. The default data directory is /usr/share/elasticsearch/data in the official Docker images that ECK uses. If you are running Elasticsearch with a large heap that is as large as the remaining space on the data volume, this can lead to a situation where Elasticsearch is no longer able to start. To avoid this scenario you have two options:

Choose a different path by setting -XX:HeapDumpPath= with the ES_JAVA_OPTS variable to a path where a volume with sufficient storage space is mounted
Resize the data volume to a sufficiently large size if your volume provisioner supports volume expansion

Capturing JVM heap dumps

edit

To take a heap dump before the JVM process runs out of memory you can execute the heap dump command directly in the Elasticsearch container:

kubectl exec $POD_NAME -- su elasticsearch -g root -c \
  '/usr/share/elasticsearch/jdk/bin/jmap -dump:format=b,file=data/heap.hprof $(pgrep java)'

If the Elasticsearch container is running with a random user ID, as for example on OpenShift, there is no need to substitute the user identity:

kubectl exec $POD_NAME -- bash -c \
  '/usr/share/elasticsearch/jdk/bin/jmap -dump:format=b,file=data/heap.hprof $(pgrep java)'

Extracting heap dumps from the Elasticsearch container

edit

To retrieve heap dumps taken by the Elasticsearch JVM or by you, as described in the previous section, you can use the kubectl cp command:

kubectl cp $POD_NAME:/usr/share/elasticsearch/data/heap.hprof ./heap.hprof


# Remove the heap dump from the running container to free up space
kubectl exec $POD_NAME -- rm /usr/share/elasticsearch/data/heap.hprof

« Elasticsearch autoscaling Security Context »