JVM settings

Here are some tips for adjusting the JVM heap size:

The operating system, persistent queue mmap pages, direct memory, and other processes require memory in addition to memory allocated to heap size.

Internal JVM data structures, thread stacks, memory mapped files and direct memory for input/output (IO) operations are all parts of the off-heap JVM memory. Memory mapped files are not part of the Logstash’s process off-heap memory, but consume RAM when paging files from disk. These mapped files speed up the access to Persistent Queues pages, a performance improvement - or trade off - to reduce expensive disk operations such as read, write, and seek. Some network I/O operations also resort to in-process direct memory usage to avoid, for example, copying of buffers between network sockets. Input plugins such as Elastic Agent, Beats, TCP, and HTTP inputs, use direct memory. The zone for Thread stacks contains the list of stack frames for each Java thread created by the JVM; each frame keeps the local arguments passed during method calls. Read on Setting the JVM stack size if the size needs to be adapted to the processing needs.

Plugins, depending on their type (inputs, filters, and outputs), have different thread models. Every input plugin runs in its own thread and can potentially spawn others. For example, each JDBC input plugin launches a scheduler thread. Netty based plugins like TCP, Beats or HTTP input spawn a thread pool with 2 * number_of_cores threads. Output plugins may also start helper threads, such as a connection management thread for each Elasticsearch output instance. Every pipeline, also, has its own thread responsible to manage the pipeline lifecycle.

To summarize, we have 3 categories of memory usage, where 2 can be limited by the JVM and the other relies on available, free memory:

Keep these memory requirements in mind as you calculate your ideal memory allocation.

Input plugins such as Elastic Agent, Beats, TCP, and HTTP allocate buffers in Java heap memory to read events from the network. Heap memory is the preferred allocation method, as it facilitates debugging memory usage problems (such as leaks and Out of Memory errors) through the analysis of heap dumps.

Before version 9.0.0, Logstash defaulted to direct memory instead of heap for this purpose. To re-enable the previous behavior Logstash provides a pipeline.buffer.type setting in logstash.yml that lets you control where to allocate memory buffers for plugins that use them.

Performance should not be noticeably affected if you switch between direct and heap. While copying bytes from OS buffers to direct memory buffers is faster, Logstash Event objects produced by these plugins are allocated on the Java Heap, incurring the cost of copying from direct memory to heap memory, regardless of the setting.

Total JVM memory allocation must be estimated and is controlled indirectly using Java heap and direct memory settings. By default, a JVM’s off-heap direct memory limit is the same as the heap size. Check out beats input memory usage. Consider setting -XX:MaxDirectMemorySize to half of the heap size or any value that can accommodate the load you expect these plugins to handle.

As you make your capacity calculations, keep in mind that the JVM can’t consume the total amount of the host’s memory available, as the Operating System and other processes will require memory too.

For a Logstash instance with persistent queue (PQ) enabled on multiple pipelines, we could estimate memory consumption using:

pipelines number * (pipeline threads * stack size + 2 * PQ page size) + direct memory + Java heap

Example

Consider a Logstash instance running 10 pipelines, with simple input and output plugins that doesn’t start additional threads, it has 1 pipelines thread, 1 input plugin thread and 12 workers, summing up to 14. Keep in mind that, by default, JVM allocates direct memory equal to memory allocated for Java heap.

The calculation results in: