WARNING: Version 1.0.1 of Filebeat has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Updating the Registry File Changes to Command Line Options »
Elastic Docs ›Filebeat Reference [1.0.1] ›Migrating from Logstash Forwarder to Filebeat

Migrating Your Configuration

edit

Migrating Your Configuration

edit

Although Filebeat is based on Logstash Forwarder, Filebeat uses YAML for its configuration file, rather than the JSON+comments language used by Logstash Forwarder. This means that you will need to migrate your existing configuration files to use the YAML syntax. Filebeat has a main configuration file called filebeat.yml, but Filebeat also accepts reading multiple configuration files from a conf.d directory and has similar restrictions to Logstash Forwarder. If you specify additional config files, you need to place them in a directory other than the directory where the main Filebeat config file resides. You specify the location of the config files by using the config_dir option to configure the path to the directory. In most cases, you can do a one-to-one conversion to create a Filebeat config file for each Logstash Forwarder config file.

Before migrating your config files, we recommend that you first read the Configuration Options section to understand the Filebeat options.

Logstash Forwarder has the option of autocompleting environment variables in the configuration file. This option currently doesn’t exist in Filebeat.

Migrating the "files" Section

edit

To migrate the files section from the Logstash Forwarder configuration, create a prospectors section in the Filebeat config file. For example, assuming that you start with this configuration in Logstash Forwarder:

  # The list of files configurations
  "files": [
    # An array of hashes. Each hash tells what paths to watch and
    # what fields to annotate on events from those paths.
    {
      "paths": [
        "/var/log/messages",
        "/var/log/*.log"
      ],

      # A dictionary of fields to annotate on each event.
      "fields": {
        "type": "syslog",
        "service": "apache",
        "zone": "us-east-1"
      }
    }, {
      # A path of "-" means stdin.
      "paths": [ "-" ],
      "fields": { "type": "stdin" }
    }, {
      "paths": [
        "/var/log/apache/httpd-*.log"
      ],
      "fields": { "type": "apache" }
    }
  ]

The equivalent prospectors section would look like this:

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      paths:
        - /var/log/messages
        - "/var/log/*.log"
      document_type: syslog 
      fields:
        service: apache
        zone: us-east-1
    -
      paths:
        - "-"
      input_type: stdin 
      document_type: stdin
    -
      paths:
        - "/var/log/apache/httpd-*.log"
      document_type: apache

The document_type option controls the output type field, which is used by the Elasticsearch output to determine the document type.

The explicit input_type option was introduced to differentiate between normal files and stdin. In the future, additional types might be supported.

As you can see, apart from the new document_type and input_type options, which were before implicitly defined via the type custom field, the remaining options can be migrated mechanically.

The Filebeat configuration gives you more control over how each prospector behaves by allowing you to configure options that were previously global in Logstash Forwarder and set them separately for each prospector. See Configuration Options.

Migrating the "network" Section

edit

Like Logstash Forwarder, Filebeat can communicate directly with Logstash. Filebeat can also insert log entries directly into Elasticsearch. This results in an output section that is a bit more complex, as you can see in the following example. You’ll find, however, that you can easily translate the Logstash part of the configuration from the equivalent Logstash Forwarder configuration.

The following snippet shows the network section of the Logstash Forwarder configuration:

  # The network section covers network configuration :)
  "network": {
    # A list of downstream servers listening for our messages.
    # logstash-forwarder will pick one at random and only switch if
    # the selected one appears to be dead or unresponsive
    "servers": [ "localhost:5043" ],

    # The path to your client ssl certificate (optional)
    "ssl certificate": "./logstash-forwarder.crt",
    # The path to your client ssl key (optional)
    "ssl key": "./logstash-forwarder.key",

    # The path to your trusted ssl CA file. This is used
    # to authenticate your downstream server.
    "ssl ca": "./logstash-forwarder.crt",

    # Network timeout in seconds. This is most important for
    # logstash-forwarder determining whether to stop waiting for an
    # acknowledgement from the downstream server. If an timeout is reached,
    # logstash-forwarder will assume the connection or server is bad and
    # will connect to a server chosen at random from the servers list.
    "timeout": 15
  }

The equivalent in Filebeat would look like this:

output:
  logstash:
    enabled: true

    # The list of downstream Logstash servers. 
    hosts:
      - localhost:5043

    tls: 
      # The path to your SSL client certificate.
      certificate: ./logstash-forwarder.crt

      # The path to your SSL client certificate key.
      certificate_key: ./logstash-forwarder.key

      # The path to your trusted SSL CA file. This is used
      # to authenticate your downstream server.
      certificate_authorities:
        - ./logstash-forwarder.crt

      # Network timeout in seconds.
      timeout: 15

When multiple hosts are defined, the default behavior in Filebeat is to pick a random host for new connections, similar to the Logstash Forwarder behavior. Filebeat can optionally do load balancing. For more details, see the loadbalance configuration option.

Note that if the tls section is missing, then TLS is disabled. TLS is automatically enabled when you add the tls section. For more information about specific configuration options, see TLS Options.

Changed Configuration File Options

edit

With the refactoring of the configuration file, the following options were removed or renamed:

Config Option Action

deadTime

deadTime was renamed to ignore_older. Filebeat keeps the files that it’s reading open until they are older than the timespan specified by ignore_older. If a file is changed, Filebeat reopens it.

netTimeout

netTimeout was removed and is replaced by the timeout option in libbeat.

log-to-syslog and syslog

Both options were removed and replaced by logging options in libbeat.

For more information about these options, see Configuration Options.

A Complete Example

edit

Let’s see a simple, but complete example of a Logstash Forwarder configuration and its equivalent for Filebeat.

Logstash Forwarder configuration:

{
  "files": [
    {
      "paths": [
        "/var/log/*.log"
      ],
      "fields": {
        "type": "syslog",
        "service": "test01"
      }
    }
  ],
  "network": {
    "servers": [ "localhost:5043" ],
  }
}

Filebeat configuration:

filebeat:
  prospectors:
    -
      paths:
        - "/var/log/*.log"
      document_type: syslog
      fields:
        service: test01
output:
  elasticsearch:
    enabled: true
    hosts: ["http://localhost:5043"]
« Updating the Registry File Changes to Command Line Options »