Use ingest pipelines for parsing
editUse ingest pipelines for parsing
editWhen you use Filebeat modules with Logstash, you can use the ingest pipelines provided by Filebeat to parse the data. You need to load the pipelines into Elasticsearch and configure Logstash to use them.
To load the ingest pipelines:
On the system where Filebeat is installed, run the setup
command with the
--pipelines
option specified to load ingest pipelines for specific modules.
For example, the following command loads ingest pipelines for the system and
nginx modules:
filebeat setup --pipelines --modules nginx,system
A connection to Elasticsearch is required for this setup step because Filebeat needs to load the ingest pipelines into Elasticsearch. If necessary, you can temporarily disable your configured output and enable the Elasticsearch output before running the command.
To configure Logstash to use the pipelines:
On the system where Logstash is installed, create a Logstash pipeline configuration
that reads from a Logstash input, such as Beats or Kafka, and sends events to an
Elasticsearch output. Set the pipeline
option in the Elasticsearch output to
%{[@metadata][pipeline]}
to use the ingest pipelines that you loaded
previously.
Here’s an example configuration that reads data from the Beats input and uses Filebeat ingest pipelines to parse data collected by modules:
input { beats { port => 5044 } } output { if [@metadata][pipeline] { elasticsearch { hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" pipeline => "%{[@metadata][pipeline]}" user => "elastic" password => "secret" } } else { elasticsearch { hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" user => "elastic" password => "secret" } } }
Set the |
See the Filebeat Modules documentation for more information about setting up and running modules.
For a full example, see Example: Set up Filebeat modules to work with Kafka and Logstash.