Logstash Moving Away from Node Protocol and Multiline Filter

By
Alvin Chen

The Logstash project has been constantly evolving and seeking out novel ways to become more robust and feature rich. At this juncture, it’s time to leave certain legacy things behind to make life easier both for Logstash users and developers. The topics to be outlined in detail here are the node protocol for Elasticsearch output and the multiline filter. Below is a brief overview of the Elastic recommended migrations for these respective components.

Discouraging the Node Protocol

The node protocol for the Elasticsearch output plugin has been a formidable pain for both Logstash users and developers. It’s tough to debug and difficult to maintain. In Logstash 2.0, the HTTP protocol has become the default as it’s very fast and possesses a much better operational experience. Additionally, a metrics API is coming soon which will circumvent any use cases where the node protocol was previously used as a way to monitor Logstash node health. Lastly, the node protocol isn't supported for communication with Shield protected Elasticsearch clusters.

Therefore, usage of the node protocol is now discouraged, and it’s strongly recommended for any current users of this protocol to migrate to the HTTP protocol in the next few months prior to the next major Logstash release. The HTTP protocol with Logstash is also the easiest way to ingest data into your clusters on Found - the best hosted Elasticsearch solution.

For any questions or concerns, please feel free to discuss in this Github issue.

Deprecating the Multiline Filter

Historically, there's been two different ways to process multiline events in Logstash. In the last couple months, the multiline codec has been strengthened on various fronts, inclusive of an enhancement which reconciled the stream identity bug. In order to mitigate confusion between the usage of the codec and filter plugins, the single-threaded multiline filter will effectively be deprecated at the Logstash 2.2 GA.

As multiline events are best processed earlier in the pipeline, it’s strongly recommended to migrate to the more scalable multiline codec in the next few months prior to the next major Logstash release. Additionally, if you’re using the awesome, lightweight Beats platform for shipping your data, this multiline processing can also be done on the edge with Filebeat.

For further details, please review this Github issue.

​Many exciting things are happening in Logstash land and we’ll be talking about them in detail at our Elastic{ON}^16 user conference in San Francisco, CA. Please join us to learn more about the Elastic stack and network with many other enthusiastic Elastic users!