Logstash 5.0.0 Breaking Changes

This section discusses the changes that you need to be aware of when migrating your application to Logstash 5.0 from the previous major release of Logstash (2.x).

These changes can impact any instance of Logstash and are plugin agnostic, but only if you are using the features that are impacted.

This file is typically located in ${LS_HOME}/config, or /etc/logstash when installed via packages. Logstash will not be able to start without this file, so please make sure to pass in --path.settings if you are starting Logstash manually after installing it via a package (RPM, DEB).

bin/logstash --path.settings /path/to/logstash.yml

The previous packages.elastic.co URL has been altered to artifacts.elastic.co. Ensure you update your repository files before running the upgrade process, or your operating system may not see the new packages.

When Logstash 5.0 is installed via DEB or RPM packages, it now uses /usr/share/logstash to install binaries. Previously it used to install in /opt/logstash directory. This change was done to make the user experience consistent with other products in the Elastic Stack.

A complete directory layout is described in Logstash Directory Layout. This will likely impact any scripts that you may have written to support installing or manipulating Logstash, such as via Puppet.

The default log severity level changed to INFO instead of WARN to match Elasticsearch. Existing logs (in core and plugins) were too noisy at the INFO level, so we auditted our log messages and switched some of them to DEBUG level.

You can use the new logstash.yml file to configure the log.level setting or continue to pass the new --log.level command line flag.

bin/logstash --log.level warn

bin/plugin has been renamed to bin/logstash-plugin. This occurred in Logstash 2.3 and it was mainly prevent PATH being polluted when other components of the Elastic Stack are installed on the same machine. Also, this provides a foundation for future change which will allow Elastic Stack packs to be installed via this script.

Logstash 5.0 also adds a remove option, which is an alias for the now-deprecated uninstall option.

As with earlier releases, the updated script allows both online and offline plugin installation. For example, to install a plugin named “my-plugin”, it’s as simple as running:

bin/logstash-plugin install my-plugin

Similar to the package changes, this is likely to impact and scripts that have been written to follow Logstash installations.

Like earlier releases of Logstash, most plugins are bundled directly with Logstash, so no additional action is required while upgrading from earlier Logstash releases. However, if you are attempting to install a non-bundled plugin, then make sure that it supports Logstash 5.0 before upgrading!

The Logstash All Plugins download option has been removed. For users previously using this option as a convenience for offline plugin management purposes (air-gapped environments), please see the Offline Plugin Management documentation page.

There were 17 plugins removed from 5.0 default bundle. These plugins can still be installed manually for use:

Some CLI Options changed in Logstash 5.0. If you were using the “long form” of the options, then this will impact the way that you launch Logstash. They were changed to match the logstash.yml format used to simplify future setup, as well as behave in the same way as other products in the Elastic Stack. For example, here’s two before-and-after examples. In Logstash 2.x, you may have run something:

bin/logstash --config my.conf --pipeline-workers 8 
bin/logstash -f my.conf -w 8 

But, in Logstash 5.0, this becomes:

bin/logstash --path.config my.conf --pipeline.workers 8 
bin/logstash -f my.conf -w 8 

The rspec script is no longer bundled with Logstash release artifacts. This script has been used previously to run unit tests for validating Logstash configurations. While this was useful to some users, this mechanism assumed that Logstash users were familiar with the RSpec framework, which is a Ruby testing framework.

Starting with Logstash 5.0, the workers setting in the Elasticsearch output plugin is no longer supported. Pipelines that specify this setting will no longer start up. You need to specify the pipeline.workers setting at the pipeline level instead. For more information about setting pipeline.workers, see Settings File.

The index template for Elasticsearch 5.0 has been changed to reflect Elasticsearch’s mapping changes. Most importantly, the subfield for string multi-fields has changed from .raw to .keyword to match Elasticsearch’s default behavior. The impact of this change to various user groups is detailed below:

Logstash is unique amongst the Elastic Stack with respect to its plugins. Unlike Elasticsearch and Kibana, which both require plugins to be targeted to a specific release, Logstash’s plugin ecosystem provides more flexibility so that it can support outside ecosystems within the same release. Unfortunately, that flexibility can cause issues when handling upgrades.

Non-standard plugins must always be checked for compatibility, but some bundled plugins are upgraded in order to remain compatible with the tools or frameworks that they use for communication. For example, the Kafka Input and Kafka Output plugins serve as a primary example of such compatibilty changes. The latest version of the Kafka plugins is only compatible with Kafka 0.10, but as the compatibility matrices show: earlier plugin versions are required for earlier versions of Kafka (e.g., Kafka 0.9).

Automatic upgrades generally lead to improved features and support, but network layer changes like those above may make part of your architecture incompatible. You should always test your Logstash configurations in a test environment before deploying to production, which would catch these kinds of issues. If you do face such an issue, then you should also check the specific plugin’s page to see how to get a compatible, older plugin version if necessary.

For example, if you upgrade to Logstash 5.0, but you want to run against Kafka 0.9, then you need to remove the bundled plugin(s) that only work with Kafka 0.10 and replace them:

bin/logstash-plugin remove logstash-input-kafka
bin/logstash-plugin remove logstash-output-kafka
bin/logstash-plugin install --version 4.0.0 logstash-input-kafka
bin/logstash-plugin install --version 4.0.1 logstash-output-kafka

The version numbers were found by checking the compatibility matrix for the individual plugins.

As described in the section above, the Kafka plugin has been updated to bring in new consumer features. In addition, to the plugin being incompatible with 0.8.x version of the Kafka broker, most of the config options have been changed to match the new consumer configurations from the Kafka Java consumer. Here’s a list of important config options that have changed:

We recommend users of the Kafka plugin to check the documentation for the latest config options.

The File Input SinceDB file is now saved at <path.data>/plugins/inputs/file location, where path.data is the path defined in the new logstash.yml file.

If you have manually specified sincedb_path as part of the configuration, this change will not affect you. If you are moving from Logstash 2.x to Logstash 5.0, and you would like to use the existing SinceDB file, then it must be copied over to path.data manually to use the save state (or the path needs to be changed to point to it).

The GeoIP filter has been updated to use MaxMind’s GeoIP2 database. Previous GeoIP version is now considered legacy by MaxMind. As a result of this, .dat version files are no longer supported, and only .mmdb format is supported. The new database will not include ASN data in the basic free database file.

Previously, when the filter encountered an IP address for which there were no results in the database, the event would just pass through the filter without modification. It will now add a _geoip_lookup_failure tag to the event which will allow for some subsequent stage of the pipeline to identify those events and perform some other operation. To simply get the same behavior as the earlier versions, just add a filter conditional on that tag which then drops the tag from the event.

With the migration to the new Event API, we have changed how you can access internal data compared to previous release. The event object no longer returns a reference to the data. Instead, it returns a copy. This might change how you perform manipulation of your data, especially when working with nested hashes. When working with nested hashes, it’s recommended that you use the field reference syntax instead of using multiple square brackets.

As part of this change, Logstash has introduced new Getter/Setter APIs for accessing information in the event object.

Examples:

Prior to Logstash 5.0, you may have used Ruby filters like so:

filter {
  ruby {
    code => "event['name'] = 'Logstash'"
  }
  ruby {
    code => "event['product']['version'] = event['major'] + '.' + event['minor']"
  }
}

The above syntax, which uses the event object as a reference, is no longer supported in Logstash 5.0. Fortunately, the change to make it work is very simple:

filter {
  ruby {
    code => "event.set('name', 'Logstash')"
  }
  ruby {
    code => "event.set('[product][version]', event.get('major') + '.' + event.get('minor'))"
  }
}

The Event API documentation describes the available syntax in great detail.