Ingest architectures

edit

We offer a variety of ingest architectures to serve a wide range of use cases and network configurations.

To ingest data into Elasticsearch, use the simplest option that meets your needs and satisfies your use case. For many users and use cases, the simplest approach is ingesting data with Elastic Agent and sending it to Elasticsearch. Elastic Agent and Elastic Agent integrations are available for many popular platforms and services, and are a good place to start.

You can host Elasticsearch on your own hardware or send your data to Elasticsearch on Elastic Cloud. For most users, Elastic Agent writing directly to Elasticsearch on Elastic Cloud provides the easiest and fastest time to value. Our hosted Elasticsearch Service is available on AWS, GCP, and Azure, and you can try it for free.

Decision tree

Data ingestion pipeline with decision tree

Ingest architecture Use when

Elastic Agent to Elasticsearch

Image showing Elastic Agent collecting data and sending to Elasticsearch

An Elastic Agent integration is available for your data source:

Elastic Agent to Logstash to Elasticsearch

Image showing Elastic Agent to Logstash to Elasticsearch

You need additional capabilities offered by Logstash:

  • enrichment between Elastic Agent and Elasticsearch
  • persistent queue (PQ) buffering to accommodate network issues and downstream unavailability
  • proxying in cases where Elastic Agents have network restrictions for connecting outside of the Elastic Agent network
  • data needs to be routed to multiple Elasticsearch clusters and other destinations depending on the content

Elastic Agent to proxy to Elasticsearch

Image showing connections between Elastic Agent and Elasticsearch using a proxy

Agents have network restrictions that prevent connecting outside of the Elastic Agent network Note that Logstash as proxy is one option.

Elastic Agent to Elasticsearch with Kafka as middleware message queue

Image showing Elastic Agent collecting data and using Kafka as a message queue enroute to Elasticsearch

Kafka is your middleware message queue:

Logstash to Elasticsearch

Image showing Logstash collecting data and sending to Elasticsearch

You need to collect data from a source that Elastic Agent can’t read (such as databases, AWS Kinesis). Check out the Logstash input plugins.

Elastic air-gapped architectures

Image showing Elastic Stack in an air-gapped environment

You want to deploy Elastic Agent and Elastic Stack in an air-gapped environment (no access to outside networks)