Migrating Your Configuration

Although Filebeat is based on Logstash Forwarder, Filebeat uses YAML for its configuration file, rather than the JSON+comments language used by Logstash Forwarder. This means that you will need to migrate your existing configuration files to use the YAML syntax. Filebeat has a main configuration file called filebeat.yml, but Filebeat also accepts reading multiple configuration files from a conf.d directory and has similar restrictions to Logstash Forwarder. If you specify additional config files, you need to place them in a directory other than the directory where the main Filebeat config file resides. You specify the location of the config files by using the config_dir option to configure the path to the directory. In most cases, you can do a one-to-one conversion to create a Filebeat config file for each Logstash Forwarder config file.

Before migrating your config files, we recommend that you first read the Configuration Options section to understand the Filebeat options.

To migrate the files section from the Logstash Forwarder configuration, create a prospectors section in the Filebeat config file. For example, assuming that you start with this configuration in Logstash Forwarder:

  # The list of files configurations
  "files": [
    # An array of hashes. Each hash tells what paths to watch and
    # what fields to annotate on events from those paths.
    {
      "paths": [
        "/var/log/messages",
        "/var/log/*.log"
      ],

      # A dictionary of fields to annotate on each event.
      "fields": {
        "type": "syslog",
        "service": "apache",
        "zone": "us-east-1"
      }
    }, {
      # A path of "-" means stdin.
      "paths": [ "-" ],
      "fields": { "type": "stdin" }
    }, {
      "paths": [
        "/var/log/apache/httpd-*.log"
      ],
      "fields": { "type": "apache" }
    }
  ]

The equivalent prospectors section would look like this:

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      paths:
        - /var/log/messages
        - "/var/log/*.log"
      document_type: syslog 
      fields:
        service: apache
        zone: us-east-1
      fields_under_root: true
    -
      input_type: stdin 
      document_type: stdin
    -
      paths:
        - "/var/log/apache2/httpd-*.log"
      document_type: apache

As you can see, apart from the new document_type and input_type options, which were before implicitly defined via the type custom field, the remaining options can be migrated mechanically.

The Filebeat configuration gives you more control over how each prospector behaves by allowing you to configure options that were previously global in Logstash Forwarder and set them separately for each prospector. See Configuration Options.

Like Logstash Forwarder, Filebeat can communicate directly with Logstash. Filebeat can also insert log entries directly into Elasticsearch. This results in an output section that is a bit more complex, as you can see in the following example. You’ll find, however, that you can easily translate the Logstash part of the configuration from the equivalent Logstash Forwarder configuration.

The following snippet shows the network section of the Logstash Forwarder configuration:

  # The network section covers network configuration :)
  "network": {
    # A list of downstream servers listening for our messages.
    # logstash-forwarder will pick one at random and only switch if
    # the selected one appears to be dead or unresponsive
    "servers": [ "localhost:5043" ],

    # The path to your client ssl certificate (optional)
    "ssl certificate": "./logstash-forwarder.crt",
    # The path to your client ssl key (optional)
    "ssl key": "./logstash-forwarder.key",

    # The path to your trusted ssl CA file. This is used
    # to authenticate your downstream server.
    "ssl ca": "./logstash-forwarder.crt",

    # Network timeout in seconds. This is most important for
    # logstash-forwarder determining whether to stop waiting for an
    # acknowledgement from the downstream server. If an timeout is reached,
    # logstash-forwarder will assume the connection or server is bad and
    # will connect to a server chosen at random from the servers list.
    "timeout": 15
  }

The equivalent in Filebeat would look like this:

output:
  logstash:
    # The Logstash hosts. 
    hosts:
      - localhost:5043

    # Network timeout in seconds.
    timeout: 15

    tls: 
      # List of root certificates for HTTPS server verifications
      certificate_authorities:
        - ./logstash-forwarder.crt

      # Certificate for TLS client authentication
      certificate: ./logstash-forwarder.crt

      # Client Certificate Key
      certificate_key: ./logstash-forwarder.key

With the refactoring of the configuration file, the following options were removed or renamed:

For more information about these options, see Configuration Options.

Let’s see a simple, but complete example of a Logstash Forwarder configuration and its equivalent for Filebeat.

Logstash Forwarder configuration:

{
  "files": [
    {
      "paths": [
        "/var/log/*.log"
      ],
      "fields": {
        "type": "syslog",
        "service": "test01"
      }
    }
  ],
  "network": {
    "servers": [ "localhost:5043" ],
  }
}

Filebeat configuration:

filebeat:
  prospectors:
    -
      paths:
        - "/var/log/*.log"
      document_type: syslog
      fields:
        service: test01
output:
  elasticsearch:
    hosts: ["http://localhost:5043"]