This documentation contains work-in-progress information for future Elastic Stack and Cloud releases. Use the version selector to view supported release docs. It also contains some Elastic Cloud serverless information. Check out our serverless docs for more details.
« Elastic Agent to proxy to Elasticsearch Elastic Agent to Logstash to Kafka to Logstash to Elasticsearch: Kafka as middleware message queue »
Elastic Docs ›Elastic Ingest Reference Architectures [8.16]

Elastic Agent to Elasticsearch with Kafka as middleware message queue

edit

Elastic Agent to Elasticsearch with Kafka as middleware message queue

edit
Image showing Elastic Agent collecting data and using Kafka as a message queue enroute to Elasticsearch
Ingest models

Elastic Agent to Logstash to Kafka to Logstash to Elasticsearch: Kafka as middleware message queue.
Logstash reads data from Kafka and routes it to Elasticsearch clusters (and/or other destinations).

Elastic Agent to Logstash to Kafka to Kafka ES Sink to Elasticsearch: Kafka as middleware message queue.
Kafka ES sink connector reads from Kafka and writes to Elasticsearch.

Use when
You are standardizing on Kafka as middleware message queue between Elastic Agent and Elasticsearch
Notes
The transformation from raw data to Elastic Common Schema (ECS) and any other enrichment can be handled by Logstash as described in Elastic Agent to Logstash (for enrichment) to Elasticsearch.

Elastic Agent with Kafka as middleware message queue architectures

edit
« Elastic Agent to proxy to Elasticsearch Elastic Agent to Logstash to Kafka to Logstash to Elasticsearch: Kafka as middleware message queue »