Breaking changes in 6.0

Version 6.0 comes with a new architecture for the internal pipeline of all Beats. This architecture refactoring is mostly internal, but one of the more visible effects is that the Spooler component of Filebeat is removed. The functionality of the Spooler was similar to the one of the publisher queue from libbeat (the code shared by all Beats), and the presence of multiple queues made the performance tuning of Filebeat more complex than it needed to be.

As a result, the following options are removed:

The first three are specific to Filebeat, while queue_size and bulk_queue_size exist in all Beats. If any of these options is set, Filebeat 6.0 will refuse to start.

Instead of the settings above, the queue.mem settings are introduced. If you had to tune the spool_size or the queue_size before, you might want to tune the queue.mem.events when upgrading. However, it is best to leave the rest of the queue.mem settings to their default values, as they are appropriate for all loads.

The publish_async option (which was deprecated since 5.3) is removed because the new pipeline already works asynchronously by default.

In versions prior to 6.0, you could enabled multiple outputs at the same time, but only of different types. For example, you were able to enable the Elasticsearch and Logstash outputs, but not two Logstash outputs. The drawback of enabling multiple outputs was that the Beats that wait for acknowledgments (Filebeat and Winlogbeat) before proceeding slowed down to the slowest output. This implication was not obvious and hindered the use cases where multiple outputs would have been useful.

As part of the pipeline re-architecture that we did for 6.0, we removed the option to enable multiple outputs at the same time. This helps with keeping the pipeline simple and with clarifying the scope of outputs in Beats.

If you require multiple outputs, you have the following options:

If you used the file or console outputs for debugging purposes, in addition to the main output, we recommend using the -d "publish" option which logs the published events in the Filebeat logs.

If you use the Logstash output to send data from Beats to Logstash, you need to update the index setting in your Logstash configuration to include the Beat version:

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Prior to 6.0, the recommended setting was:

    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

The index templates that ship with 6.0 are applied to new indexes that match the pattern [beat]-[version]-*. You must update your Logstash config, or the templates will not be applied.

The document_type setting, from the prospector configuration, was removed because the _type concept is being removed from Elasticsearch. Instead of the document_type setting, you can use a custom field.

This has led also to the rename of the input_type configuration setting to type. This change is backwards compatible because the old setting still works. However, the input_type output field was renamed to prospector.type.

The default startup behaviour (based on the included sample configuration) of Filebeat was to read all the files matching the /var/log/*.log pattern. Starting with version 6.0, Filebeat doesn’t read any files in its default configuration. However, you can easily enable the system module, for example with a CLI flag:

filebeat --modules=system

The outputs.elasticsearch.template.* settings have been moved under setup.template.*, but are otherwise unchanged.

The dashboards.* settings have been moved under setup.dashboards.*.

The Filebeat deprecated options force_close_files and close_older are removed.

The import_dashboards program, used to load the Kibana dashboards in previous versions of Beats, is replaced by the setup command. For example, the following command:

./scripts/import_dashboards -user elastic -pass YOUR_PASSWORD

Can be replaced with:

./filebeat setup -E "output.elasticsearch.username=elastic" -E "output.elasticsearch.password=YOUR_PASSWORD"

Note that the -E flags are only required if the Elasticsearch output is not already configured in the configuration file.

Besides the change in the commands, it’s important to note that loading the Kibana dashboards works differently in the 6.0 version of the stack. Prior to 6.0, the dashboards were inserted directly in the .kibana Elasticsearch index. Starting with 6.0, the Beats use a Kibana server API. This means that the Beat that loads the dashboards needs direct access to Kibana and that the Kibana URL needs to be set. The option to set the Kibana URL is setup.kibana.host, which you can set in the configuration file or via the -E CLI flag:

./filebeat setup -E "setup.kibana.host=http://kibana-host:5601"

The default value for the Kibana host is localhost:5601.

The "local" processors, which are configured at the module level, used to be called filters in Metricbeat, but were offering similar functionality with the global processors. A notable difference between the two was that the filters accessed fields relatively to the metricset (for example, mount_point), while the processors referred to fields by their fully qualified name (for example, system.filesystem.mount_point).

Starting with version 6.0, the filters are renamed to processors and they can access the fields only by using the fully qualified names.

Prior to 6.0, Metricbeat and Packetbeat were compiled using Cgo, while Filebeat, Winlogbeat, and Heartbeat were compiled using the pure Go compiler. One of the side-effects of compiling with Cgo is that libc is dynamically compiled. Starting with 6.0, all the Beats are compiled using Cgo and therefore dynamically compiled against libc. This can reduce the portability of the binaries, but none of the supported platforms is affected.