WARNING: Version 1.2 of Filebeat has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
Migrating Your Configuration
editMigrating Your Configuration
editAlthough Filebeat is based on Logstash Forwarder, Filebeat uses YAML for its configuration
file, rather than the JSON+comments language used by Logstash Forwarder. This means that you
will need to migrate your existing configuration files to use the YAML syntax. Filebeat has a main
configuration file called filebeat.yml
, but Filebeat also accepts reading
multiple configuration files from a conf.d
directory and has similar restrictions to Logstash Forwarder.
If you specify additional config files, you need to place them in a directory other than the directory
where the main Filebeat config file resides. You specify the location of the config files by using the
config_dir
option to configure the path to the directory. In most cases, you can do a one-to-one
conversion to create a Filebeat config file for each Logstash Forwarder config file.
Before migrating your config files, we recommend that you first read the Configuration Options section to understand the Filebeat options.
Migrating the "files" Section
editTo migrate the files
section from the Logstash Forwarder configuration, create a prospectors
section in the Filebeat config file. For example, assuming that you start
with this configuration in Logstash Forwarder:
# The list of files configurations "files": [ # An array of hashes. Each hash tells what paths to watch and # what fields to annotate on events from those paths. { "paths": [ "/var/log/messages", "/var/log/*.log" ], # A dictionary of fields to annotate on each event. "fields": { "type": "syslog", "service": "apache", "zone": "us-east-1" } }, { # A path of "-" means stdin. "paths": [ "-" ], "fields": { "type": "stdin" } }, { "paths": [ "/var/log/apache/httpd-*.log" ], "fields": { "type": "apache" } } ]
The equivalent prospectors
section would look like this:
filebeat: # List of prospectors to fetch data. prospectors: # Each - is a prospector. Below are the prospector specific configurations - paths: - /var/log/messages - "/var/log/*.log" document_type: syslog fields: service: apache zone: us-east-1 fields_under_root: true - input_type: stdin document_type: stdin - paths: - "/var/log/apache2/httpd-*.log" document_type: apache
The |
|
The explicit |
As you can see, apart from the new document_type
and input_type
options,
which were before implicitly defined via the type
custom field, the remaining
options can be migrated mechanically.
The Filebeat configuration gives you more control over how each prospector behaves by allowing you to configure options that were previously global in Logstash Forwarder and set them separately for each prospector. See Configuration Options.
Migrating the "network" Section
editLike Logstash Forwarder, Filebeat can communicate directly with Logstash.
Filebeat can also insert log entries directly
into Elasticsearch. This results in an output
section that is a bit more complex, as
you can see in the following example. You’ll find, however, that you can easily
translate the Logstash part of the configuration from the equivalent Logstash Forwarder
configuration.
The following snippet shows the network
section of the Logstash Forwarder configuration:
# The network section covers network configuration :) "network": { # A list of downstream servers listening for our messages. # logstash-forwarder will pick one at random and only switch if # the selected one appears to be dead or unresponsive "servers": [ "localhost:5043" ], # The path to your client ssl certificate (optional) "ssl certificate": "./logstash-forwarder.crt", # The path to your client ssl key (optional) "ssl key": "./logstash-forwarder.key", # The path to your trusted ssl CA file. This is used # to authenticate your downstream server. "ssl ca": "./logstash-forwarder.crt", # Network timeout in seconds. This is most important for # logstash-forwarder determining whether to stop waiting for an # acknowledgement from the downstream server. If an timeout is reached, # logstash-forwarder will assume the connection or server is bad and # will connect to a server chosen at random from the servers list. "timeout": 15 }
The equivalent in Filebeat would look like this:
output: logstash: # The Logstash hosts. hosts: - localhost:5043 # Network timeout in seconds. timeout: 15 tls: # List of root certificates for HTTPS server verifications certificate_authorities: - ./logstash-forwarder.crt # Certificate for TLS client authentication certificate: ./logstash-forwarder.crt # Client Certificate Key certificate_key: ./logstash-forwarder.key
When multiple hosts are defined, the default behavior in Filebeat is to pick a random host for new connections, similar to the Logstash Forwarder behavior. Filebeat can optionally do load balancing. For more details, see the loadbalance configuration option. |
|
Note that if the |
Changed Configuration File Options
editWith the refactoring of the configuration file, the following options were removed or renamed:
Config Option | Action |
---|---|
|
|
|
|
|
Both options were removed and replaced by logging options in libbeat. |
For more information about these options, see Configuration Options.
A Complete Example
editLet’s see a simple, but complete example of a Logstash Forwarder configuration and its equivalent for Filebeat.
Logstash Forwarder configuration:
{ "files": [ { "paths": [ "/var/log/*.log" ], "fields": { "type": "syslog", "service": "test01" } } ], "network": { "servers": [ "localhost:5043" ], } }
Filebeat configuration:
filebeat: prospectors: - paths: - "/var/log/*.log" document_type: syslog fields: service: test01 output: elasticsearch: hosts: ["http://localhost:5043"]