Elastic Agent: Flexibility to send and process any data, anywhere

Starting from Elastic Agent 8.16, with “output per integration,” users can send data from any supported data source to any supported destination. Whether it's directly to regional or business line-specific Elasticsearch clusters or through intermediaries like Logstash and Kafka, the process seamlessly scales to the largest enterprises. Deploying a single agent to collect observability and security data has never been easier.

With the fine-grained control of outputs per integration, including Logstash in your ingest architecture is no longer an all-or-nothing decision. You can now include Logstash as needed for specific data sources to implement redaction, enrichment, and other capabilities unique to Logstash. 

For example, if you’re using the new Logstash integration filter with your Elastic integrations, you can now send only the data that you want Logstash to process while sending the rest of your data sources straight to Elasticsearch. Reducing the workload on Logstash enables a smaller ingest footprint, less time managing nodes, and higher reliability.

Kafka — a distributed, highly resilient, and scalable streaming platform — is a very popular tool for processing real-time streaming data. Kafka’s producer and consumer model allows operators to set up different pipelines for logs and metrics or security and observability or sensitive and confidential information. Elastic Agents fit right into this model with the ability to write to a specific Kafka topic, and with the release of 8.16, Elastic Agents can dynamically choose Kafka topics.  

Consider a multicloud enterprise with applications deployed across redundant regions and cloud providers. Managing ingest configuration per region adds administrative overhead, but managing ingest globally means costly cloud egress fees as data traverses zones and providers.

With output per integration, you can maintain global or cloud-specific Elastic Agent control planes while keeping data local to the zone or region, saving hundreds of thousands of dollars in cloud networking costs.

Consider an enterprise that needs to ensure data collected in a region stays within that region (data sovereignty). Fleet simplifies the process by allowing operators to quickly assign an output to a group of Elastic Agents through agent policies.

In many deployments, users need to add local context to the data ingested into their platform. This embedded context helps operators uniquely identify data for further processing, routing, and even creating context-specific dashboards, such as those for a specific tenant.

With the release of version 8.15, Elastic Agent users can now add custom [field:value] pairs at the policy level. These custom fields are embedded into all data collected by agents under that policy. As logs and events move through the pipeline, the added fields can be used to enhance or route data effectively. You can find more information about this feature in our documentation.

In summary, Elastic Agent 8.16 offers unparalleled flexibility in data collection and output management, allowing users to send data precisely where it's needed. With features like output per integration, optional Logstash processing, dynamic Kafka forwarding, and enhanced enterprise policy management, Elastic Agent streamlines complex data workflows while optimizing resource use. Whether it's reducing cloud egress fees or ensuring compliance with data sovereignty regulations, Elastic Agent empowers developers and operators to build scalable, resilient, and efficient data pipelines, making it an invaluable tool for modern enterprises.