Easy, comprehensive Logstash monitoring with Elastic Agent

Logstash is a powerful tool for ingesting, transforming, and shipping data from various sources. Visibility into Logstash is critical for optimizing performance and troubleshooting issues related to data ingestion. We’ve greatly improved the Logstash integration to display the status of your Logstash nodes and pipelines at a glance. The integration is now powered by Elastic Agent, which queries Logstash monitoring APIs for data that populates managed dashboards. 

This blog will guide you through the visualizations available from the integration, how to configure and install them, and the underlying APIs that provide the data. See the health of your Logstash nodes at a glance with the updated integration.

For the performance of a single node, the Node Overview dashboard provides details on system and process health. This helps to determine if an issue is localized to an individual host or if it is pervasive throughout a deployment.

Pipelines are where the work in Logstash really occurs, and we have no shortage of insights related to pipeline performance. The Pipeline Overview dashboard provides high-level details on pipeline activity across your entire deployment to quickly surface problems and assess performance. Broken down by pipeline, you can see the average time an event takes to process, which is how long it takes pushing to queue. Items that are outside the allotted tolerance are bolded in red to quickly attract attention and direct users to take action.

As you drill down into individual pipelines, you’ll see details on worker utilization and processing over time. This is invaluable in determining the cause of problems. Isolating spikes in time makes it easier to correlate with configuration changes or external events. 

But you shouldn’t stop at pipelines. A huge benefit of using Logstash is the vast ecosystem of plugins. That strength does introduce variability, which can make troubleshooting more difficult. Thankfully, the integration addresses that problem by providing plugin details by type, enabling users to drill down further.

The health report dashboards are the newest addition, and the amount of information they can quickly convey will make every Logstash operator’s life easier. One quick look is all you need to see if there are problems with your Logstash deployment.

Pipeline health provides more detailed data as well as actionable insights on troubleshooting steps. 

Installing the integration can be done in minutes, and the data you get will save you hours of troubleshooting. Here are the steps to install the integration and deploy agents to your Logstash instances that will collect and transform the data from Logstash monitoring APIs.  

1. Add the Logstash integration to monitor your deployment
From the integrations page, search for Logstash. Click Add Logstash, and you’ll be guided through the configuration options on how an Elastic Agent can collect monitoring data from your instance.

Be sure to select Metrics (Elastic Agent) to get the most data and our newest dashboards. You can specify what data to collect and the API polling intervals or stick with our defaults. Health Reporting is disabled by default, but you’ll want to enable it to get the most out of the integration. If you have modified your Logstash configuration to listen on a different port, specify it here.

2. Configure and install Elastic Agents on Logstash nodes

You will likely be deploying this to new agents. Create a name that is meaningful, and then save and continue. If you already have agents monitoring your Logstash deployment, you can add this policy to existing hosts.

You will be prompted to add Elastic Agent to your Logstash nodes.

Choose Add Elastic Agent to your hosts, and you’ll be guided through Fleet enrollment, agent installation, and verification.

Once you’ve verified that agents are communicating with Fleet, the data will start flowing in and populate the dashboards.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.