Logstash 7.9.0 Release Notes

This release is the first step toward Elastic Common Schema (ECS) support in Logstash. With 7.9, you can configure the Elasticsearch output plugin to manage index templates that are compatible with the Elastic Common Schema (ECS). The ECS compatibility setting in the Elasticsearch output plugin makes this possible.

See Compatibility with the Elastic Common Schema (ECS) in the Elasticsearch output plugin docs for more information.

With this release, we’ve continued expanding support for Elasticsearch API keys. Support for API keys in the Elasticsearch output plugin arrived in Logstash 7.8.0. Logstash 7.9.0 introduces support for Elasticsearch API keys in the Elasticsearch input plugin, the Elasticsearch filter plugin, and Logstash monitoring and management.

Check out Grant access using API keys for more information about using API keys with Logstash and Elasticsearch. Implementation details are in #11953.

Logstash runs on arm machines! We have tested Logstash against arm64, and we are looking to make docker and other images available soon.

ARM artifacts are not yet supported for production, and we’re offering them as "experimental" to early adopters.

We replaced the deprecated Java client library for the Elastic App Search output plugin with the Ruby client library, and expanded integration testing. These changes provide a foundation for expanding App Search integration and quality assurance in future releases.

We’ve enhanced persistent queues to better manage exceptions and error handling which could sometimes result in a LockException when the queue file lock was not properly released. Under some conditions, a complex pipeline that is slower to initialize could be recreated when it was not done initializing, causing a LockException. Implementation details are in #12023.

These changes result in better stability of persistent queues.

Worker threads were not correctly monitored for a worker loop exception resulting in a complete logstash crash upon any exception even when multiple pipelines are running. Now only the failed pipeline is terminated. If pipeline reloading is enabled, you can edit the config and have the failed pipeline reloaded. Implementation details are in #12019 and #12038.

This release contains several optimizations to pipeline compilation, an essential step of the pipeline initialization process. These changes significantly improve startup and pipeline-restart performance for complex pipelines. (For technical details, check out this PR: #12060.)

From our tests in three different pipelines with eight workers each, we have seen times decrease from 9 - 28 minutes to around 1 minute.

To aid the development of pipelines, especially the performance impact of compilation, Logstash now reports the time taken to compile each pipeline as a log entry such as:

[2020-08-12T14:10:29,388][INFO ][logstash.javapipeline  ][main] Pipeline Java execution initialization time {"seconds"=>0.7}