- Filebeat Reference: other versions:
- Overview
- Getting Started With Filebeat
- Step 1: Install Filebeat
- Step 2: Configure Filebeat
- Step 3: Configure Filebeat to use Logstash
- Step 4: Load the index template in Elasticsearch
- Step 5: Set up the Kibana dashboards
- Step 6: Start Filebeat
- Step 7: View the sample Kibana dashboards
- Quick start: modules for common log formats
- Repositories for APT and YUM
- Setting up and running Filebeat
- Upgrading Filebeat
- How Filebeat works
- Configuring Filebeat
- Specify which modules to run
- Configure inputs
- Manage multiline messages
- Specify general settings
- Load external configuration files
- Configure the internal queue
- Configure the output
- Set up index lifecycle management
- Load balance the output hosts
- Specify SSL settings
- Filter and enhance the exported data
- Parse data by using ingest node
- Enrich events with geoIP information
- Set up project paths
- Set up the Kibana endpoint
- Load the Kibana dashboards
- Load the Elasticsearch index template
- Configure logging
- Use environment variables in the configuration
- Autodiscover
- YAML tips and gotchas
- Regular expression support
- HTTP Endpoint
- filebeat.reference.yml
- Beats central management
- Modules
- Exported fields
- Alias fields
- Apache2 fields
- Auditd fields
- Beat fields
- Cloud provider metadata fields
- Docker fields
- elasticsearch fields
- haproxy fields
- Host fields
- Icinga fields
- IIS fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- mongodb fields
- MySQL fields
- Nginx fields
- Osquery fields
- PostgreSQL fields
- Redis fields
- System fields
- Traefik fields
- Monitoring Filebeat
- Securing Filebeat
- Troubleshooting
- Migrating from Logstash Forwarder to Filebeat
- Contributing to Beats
Manage multiline messages
editManage multiline messages
editThe files harvested by Filebeat may contain messages that span multiple
lines of text. For example, multiline messages are common in files that contain
Java stack traces. In order to correctly handle these multiline events, you need
to configure multiline settings in the filebeat.yml file to specify
which lines are part of a single event.
If you are sending multiline events to Logstash, use the options described here to handle multiline events before sending the event data to Logstash. Trying to implement multiline event handling in Logstash (for example, by using the Logstash multiline codec) may result in the mixing of streams and corrupted data.
Also read YAML tips and gotchas and Regular expression support to avoid common mistakes.
Configuration options
editYou can specify the following options in the filebeat.inputs section of
the filebeat.yml config file to control how Filebeat deals with messages
that span multiple lines.
The following example shows how to configure Filebeat to handle a multiline message where the first line of the message begins with a bracket ([).
multiline.pattern: '^\[' multiline.negate: true multiline.match: after
Filebeat takes all the lines that do not start with [ and combines them with the previous line that does. For example, you could use this configuration to join the following lines of a multiline message into a single event:
[beat-logstash-some-name-832-2015.11.28] IndexNotFoundException[no such index] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77) at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteIndexAction.java:75)
-
multiline.pattern -
Specifies the regular expression pattern to match. Note that the regexp patterns supported by Filebeat
differ somewhat from the patterns supported by Logstash. See Regular expression support for a list of supported regexp patterns.
Depending on how you configure other multiline options, lines that match the specified regular expression are considered
either continuations of a previous line or the start of a new multiline event. You can set the
negateoption to negate the pattern. -
multiline.negate -
Defines whether the pattern is negated. The default is
false. -
multiline.match -
Specifies how Filebeat combines matching lines into an event. The settings are
afterorbefore. The behavior of these settings depends on what you specify fornegate:Setting for negateSetting for matchResult Example pattern: ^bfalseafterConsecutive lines that match the pattern are appended to the previous line that doesn’t match.
falsebeforeConsecutive lines that match the pattern are prepended to the next line that doesn’t match.
trueafterConsecutive lines that don’t match the pattern are appended to the previous line that does match.
truebeforeConsecutive lines that don’t match the pattern are prepended to the next line that does match.
The
aftersetting is equivalent topreviousin Logstash, andbeforeis equivalent tonext. -
multiline.flush_pattern - Specifies a regular expression, in which the current multiline will be flushed from memory, ending the multiline-message.
-
multiline.max_lines -
The maximum number of lines that can be combined into one event. If
the multiline message contains more than
max_lines, any additional lines are discarded. The default is 500. -
multiline.timeout - After the specified timeout, Filebeat sends the multiline event even if no new pattern is found to start a new event. The default is 5s.
On this page