Improving the event queue in Elastic Agent and Beats

Events received by Elastic Agent and Beats are stored in a queue while they’re being sent to the output. The queue’s configuration makes a big difference in how much memory is needed to store those events. Prior to 8.13, this configuration involved some easily misunderstood parameters that were a frequent source of misconfigurations.

Let's review some of those tuning parameters and their roles.

When an output worker is ready to send data, it requests a batch of events from the internal queue. The size of this request is controlled by bulk_max_size, an important output tuning parameter. If bulk_max_size is 100, then the queue will try to provide 100 events for the output worker to send.

The queue also has a flush.timeout parameter. When this is zero, the queue will return the events immediately, even if it doesn't have enough. In our example, if 100 events are requested but there are only 50 in the queue, then the output worker will get 50 events. But when the flush timeout is positive, the queue will instead wait for up to the specified timeout to collect more events.

But hold on: let's say we set a flush timeout of 5 seconds and request 100 events. You might expect that the queue will return 100 events immediately if it has them or else it will delay for up to 5 seconds trying to get to 100. And starting in version 8.13, you'd be right. But that isn't how the old queue behaved! The queue didn’t wait to fill an output request — it waited to fill an internal queue buffer, which could be a completely different size.

The size of the internal queue buffers was controlled by flush.min_events, a parameter that looks very similar to bulk_max_size and is often confused for it but can have very different effects.

Here are some performance problems in the old queue that are fixed by the changes in 8.13.

We ended up rewriting the queue to remove the chain of internal buffers entirely. Instead, we now use a single fixed buffer, wrapping around the end as needed. Events are also copied out of this shared buffer to assemble output batches. flush.timeout now works exactly the way most people expected it to: the queue will immediately return the requested number of events if they're available or else it will wait up to the specified timeout to fill the request (but only the current request, not some larger internal limit!).

This refactor required a compromise that the original queue tried to avoid — event batches are no longer a single contiguous sequence in memory. But in exchange, we get huge benefits:

• The memory problem in example one is gone. As soon as the output confirms a batch of events has been processed, the queue can release its memory no matter where it falls relative to other event batches.

• The latency problem in example two is gone. If the queue has enough events, it will return them. It doesn't matter where they fall in the event sequence.

• The batch size problem in example three is gone. Since events no longer need to come from a contiguous buffer, we don't need to split batches based on internal memory boundaries.

flush.min_events is now a legacy parameter which, for backwards compatibility, specifies a global maximum on the size of an event batch. If you're using a performance preset, you can ignore this parameter entirely, but custom queue configurations are now recommended to set it to a large value and use bulk_max_size to control batch size. As long as flush.min_events is large enough, there's no longer any performance advantage to changing it.

In our internal benchmarks, we saw significant memory savings for all presets (these differences were measured with Filebeat using a filestream input on structured json events):

All presets also show a 3% reduction in the CPU cost per event.

By defining built-in presets for common performance goals, we took a lot of the guesswork out of configuring Elastic Agent and Beats. Now we're seeing the follow-up benefits, as this gives us a set of common configurations to target in our optimization work. Expect these improvements to continue in version 8.14 and beyond!

What else is new in Elastic 8.13? Check out the 8.13 announcement post to learn more >>