News

Logstash 1.5.0 GA released

By
Suyog Rao

We are excited to announce the GA release of Logstash 1.5.0! 500 plus commits and 10 months in the making, this is one of our biggest releases. A big thank you to all our users for the many contributions, feedback, GitHub issues, and for trying out the pre-releases. Jump to our downloads page to get going, or check the changelog for details.

Highlights

The main themes of 1.5.0 are plugin management, performance improvements, and Apache Kafka integration. Here are the highlights:

Plugins Separation

Logstash has a rich collection of over 165 plugins (inputs, filters, outputs, and codecs). With 1.5.0, we are taking a step closer to making plugin management even better for our users. We have added infrastructure to easily install, update, and remove plugins on top of Logstash.

For example, to install the S3 output plugin: 

$LS_HOME/bin/plugin install logstash-output-s3

This blog has all the scoop about plugin changes.

New plugins

As we iterated on 1.5.0, the feedback we received for plugin changes was overwhelming. This is evident with the contribution of many new plugins -- Heartbeat input, CouchDB changes input, Slack input, RSS input, JMX input just to name a few. Check out how easy it is to create and publish plugins!

Performance Improvements

Logstash 1.5.0 is much faster. Let's highlight two areas where performance has gone way up:

Grok Filter: In this release, we have increased the throughput of the popular grok filter in some patterns by 100%. Put another way, you can process more data through Logstash when using the grok filter. In our benchmark testing, we compared throughput in 1.5.0 and 1.4.2 by processing 6.9 million entries of Apache Web access log lines using the COMBINEDAPACHELOG grok pattern. Throughput in 1.5.0 increased from 34K events per second (eps) to 50K eps. Both tests were run on an eight-core machine with eight worker threads in Logstash. These tests we run with a single grok filter and measured throughput of events processed in the pipeline using a stdin input and stdout output. Please note that overall performance will vary with hardware and Logstash configuration used.

JSON Serialization/Deserialization: We implemented JSON serialization and deserialization using the JrJackson library, which improved the throughput by over 100%. In our previously mentioned performance tests, we sent 500,000 JSON events (1.3KB in size) and measured a throughput increase from 16K eps to 30K eps. With events 45KB in size, throughput increased from 850 eps to 3.5K eps.

Improved Security

We have improved the security of the Elasticsearch output, input, and filter by adding authentication and transport encryption support. For instance, with the HTTP protocol you can configure SSL/TLS to enable encryption and HTTP basic authentication to provide a username and password while making requests. These capabilities will enable Logstash to natively integrate with the Elastic's Shield security product.

Apache Kafka Integration

In scaling Logstash deployments, Apache Kafka can be used as an intermediate message buffer to store data between the shipping instances and indexing instances. In 1.5.0, we have added built-in support for the Logstash Kafka input and output plugin that was originally developed by Joseph Lawson. We added the Apache Avro codec and support to easily plug in your serialization/deserialization mechanism. All this makes it easy to consume events stored in Kafka, enrich them, and analyze them using Elasticsearch, Logstash, and Kibana.

Windows Experience

This release made many improvements for running Logstash and plugin-related infrastructure on Windows, which was degraded since the 1.4.2 release. We resolved issues related to initial setup and upgrade, and fixed important bugs in the file input plugin.

Documentation

Previously, Logstash documentation was hosted on logstash.net, which made it cumbersome to find information when working with the rest of the ELK stack. We have moved documentation for 1.5.0 and all future releases to the Elastic website under the Logstash Guide.

Fixes/Enhancements

1.5.0 contains a number of important enhancements and bug fixes. A detailed list can be found in the changelog.

What's next?

Though the team has been focused on 1.5.0 release in the last few months, it hasn't stopped us from plotting the next course for Logstash. We are really excited about the feedback received from the Logstash community survey and can't wait to get started on the Logstash 2.0 roadmap. Stay tuned on our progress with our weekly updates!

Feedback

Download 1.5.0 today and let us know what you think. We hang out in IRC, so join us in #logstash on Freenode. You can also provide us contributions, suggestions, and feedback through GitHub, or ask questions on our forum. You can also tweet us at @elastic