IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Elastic Agent to Logstash to Elasticsearch: Logstash as a proxy Elastic Agent to proxy to Elasticsearch »
Elastic Docs ›Elastic Ingest Reference Architectures [8.13] ›Elastic Agent to Logstash to Elasticsearch

Elastic Agent to Logstash for routing to multiple Elasticsearch clusters and additional destinations

edit

Elastic Agent to Logstash for routing to multiple Elasticsearch clusters and additional destinations

edit
Image showing Elastic Agent collecting and routing data to multiple destinations
Ingest model
Elastic Agent to Logstash to Elasticsearch clusters and/or additional destinations
Use when
Data collected by Elastic Agent needs to be routed to different Elasticsearch clusters or non-Elasticsearch destinations depending on the content
Example

Let’s take an example of a Windows workstation, for which we are collecting different types of logs using the System and Windows integrations. These logs need to be sent to different Elasticsearch clusters and to S3 for backup and a mechanism to send it to other destinations such as different SIEM solutions. In addition, the Elasticsearch destination is derived based on the type of datastream and an organization identifier.

In such use cases, agents send the data to Logstash as a routing mechanism to different destinations. Note that the System and Windows integrations must be installed on all Elasticsearch clusters to which the data is routed.

Sample config 
input {
  elastic_agent {
    port => 5044
  }
}
filter {
  translate {
    source => "[http][host]"
    target => "[@metadata][tenant]"
    dictionary_path => "/etc/conf.d/logstash/tenants.yml"
  }
}
output {
  if [@metadata][tenant] == "tenant01" {
    elasticsearch {
      cloud_id => "<cloud id>"
      api_key => "<api key>"
    }
  } else if [@metadata][tenant] == "tenant02" {
    elasticsearch {
      cloud_id => "<cloud id>"
      api_key => "<api key>"
    }
  }
}

Resources

edit

Info on configuring Elastic Agent:

Info on Logstash and Logstash outputs:

Info on Elasticsearch:

« Elastic Agent to Logstash to Elasticsearch: Logstash as a proxy Elastic Agent to proxy to Elasticsearch »