Using Elastic Agent Performance Presets in 8.12

The default tuning parameters used with Elastic Agent today have been in place since the beginning of Beats and were picked to ensure that data gets into Elasticsearch® as quickly as possible for analysis. Historically, tweaking performance of Elastic Agent and Beats involved deep knowledge, and benchmarking them for an optimal combination requires even more expertise.

With 8.12, we’ve made tuning Elastic Agent and Beats a thing of the past for the vast majority of customers. We’ve introduced new default settings for Elastic Agent and Beats that improve throughput by up to 50%, reduce memory usage by 10%, reduce concurrent connections to Elasticsearch by up to 80%, and reduce Disk I/O in Elasticsearch from Beats/Agent requests by up to 50%.

If the new defaults don’t meet your needs, or perhaps you need more throughput or want to scale to a larger deployment of agents, the next step is not tuning parameters in a YAML file; instead, new Performance Presets take the best practices from our trusted customers, field teams, and performance test beds and allow you to pick the preset that best suits your needs, effectively offering a tailored configurations to meet your diverse performance requirements. Let's explore the four presets and the new Custom option and understand which one suits your needs best:

From 8.12 onward, the Balanced preset is the default preset and optimizes Elastic Agent to achieve a reasonable level of throughput and resource utilization, making it the preferred choice for a wide array of Elastic Agent use cases.

For those of you who are familiar with the various settings available within Beats, here are the exact changes happening to the default values in 8.12. These new defaults increase the Events Per Second throughput (EPS) by 25%.

Please note: These settings may change in future versions as we continue to tune these for the best performance.

If you’re deploying Elastic Agent to tens of thousands of systems, the Optimized for Scale preset fine-tunes Elastic Agent for gathering lower volumes of host data like logs or metrics but from tens or hundreds of thousands of devices. It takes the best parts of the Balanced preset, but trades a short delay in data ingestion for a total 30x reduction in open connections to Elasticsearch over the defaults. This makes a big difference if you’re using a reverse proxy or load balancer in front of Elasticsearch. This setting’s EPS performance is fairly similar to that of the Balanced setting.

The Optimized for Latency preset reduces the time between data ingestion and processing, making it suitable for low throughput but real-time analytics and applications that demand low latency. This preset closely mirrors the pre-8.12 Elastic Agent default settings and is the perfect fallback for any issues encountered testing the newer presets.

In your Elastic environment, it’s important to note that these presets only impact Agents on version 8.12+. Older versions of Agent will continue to use their existing settings.

While presets are designed to simplify the tuning process for Elastic Agent, the Custom option allows the user to have more granular control over performance. You can still refer to the old guidance provided by Elastic, which is still relevant on Agent as the queue.mem.events are now also configurable. The old guidance, available here, offers practical examples and insights on tuning Elastic Beats performance, including batch size, worker count, and more.

To provide further guidance, the table in our public documentation shows the effects on EPS throughput performance as a function of the performance of the Balanced preset, for various permutations of these tuning parameters, for users who require more granular control. These parameters can be set when the Custom preset is chosen.

In conclusion, Elastic's new Performance Presets provide a convenient way to fine-tune Elastic Agent for different performance requirements. Whether you need a balanced performance, high throughput, scalability, or low latency, these presets offer optimized configurations to meet your needs. Happy tuning!

What else is new in Elastic 8.12? Check out the 8.12 announcement post to learn more >>