Use ingest pipelines for parsing

When you use Filebeat modules with Logstash, you can use the ingest pipelines provided by Filebeat to parse the data. You need to load the pipelines into Elasticsearch and configure Logstash to use them.

To load the ingest pipelines:

On the system where Filebeat is installed, run the setup command with the --pipelines option specified to load ingest pipelines for specific modules. For example, the following command loads ingest pipelines for the system and nginx modules:

filebeat setup --pipelines --modules nginx,system

A connection to Elasticsearch is required for this setup step because Filebeat needs to load the ingest pipelines into Elasticsearch. If necessary, you can temporarily disable your configured output and enable the Elasticsearch output before running the command.

To configure Logstash to use the pipelines:

On the system where Logstash is installed, create a Logstash pipeline configuration that reads from a Logstash input, such as Beats or Kafka, and sends events to an Elasticsearch output. Set the pipeline option in the Elasticsearch output to %{[@metadata][pipeline]} to use the ingest pipelines that you loaded previously.

Here’s an example configuration that reads data from the Beats input and uses Filebeat ingest pipelines to parse data collected by modules:

input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create" 
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create"
      user => "elastic"
      password => "secret"
    }
  }
}

See the Filebeat Modules documentation for more information about setting up and running modules.

For a full example, see Example: Set up Filebeat modules to work with Kafka and Logstash.