Logstash-to-Logstash communication
editLogstash-to-Logstash communication
editLogstash-to-Logstash communication is available if you need to have one Logstash instance communicate with another Logstash instance. Implementing Logstash-to-Logstash communication can add complexity to your environment, but you may need it if the data path crosses network or firewall boundaries. However, we suggest you don’t implement unless it is strictly required.
If you are looking for information on connecting multiple pipelines within one Logstash instance, see Pipeline-to-pipeline communication.
Logstash-to-Logstash communication can be achieved in one of two ways:
Lumberjack-Beats considerations
Lumberjack output to Beats input has been our standard approach for Logstash-to-Logstash communication, and may still be the best option for more robust use cases. Before you implement the Lumberjack to Beats configuration, keep these points in mind:
- Lumberjack to Beats provides high availability, but does not provide load balancing. The Lumberjack output plugin allows defining multiple output hosts for high availability, but instead of load-balancing between all output hosts, it falls back to one host on the list in the case of failure.
- If you need a proxy between the Logstash instances, TCP proxy is the only option.
- There’s no explicit way to exert back pressure back to the beats input.
Ready to see more configuration details? See Logstash-to-Logstash: Lumberjack output to Beats input.
This approach relies on the use of http output to http input plugins. Take these considerations into account before you implement:
- HTTP does not provide built-in high availability. You will need to implement your own load balancer in between the HTTP output and the HTTP input.
- If you need a proxy between the Logstash instances, you can use any HTTP proxy.
- The HTTP input adds connection information to events, and this may be data you don’t want.
For now, http output to http input with manual configuration may be the best path forward if these limitations don’t apply to your use case.
In the future, we may replace the implementation of Logstash-to-Logstash with a purpose-build HTTP implementation, which would deprecate the use of Lumberjack and Beats, or the use of the HTTP Input and Output plugins.