IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Monitoring Logstash Collect Logstash monitoring data using legacy collectors »
Elastic Docs ›Logstash Reference [8.3] ›Monitoring Logstash

Collect Logstash monitoring data with Metricbeat

edit

Collect Logstash monitoring data with Metricbeat

edit

You can use Metricbeat to collect data about Logstash and ship it to the monitoring cluster. The benefit of Metricbeat collection is that the monitoring agent remains active even if the Logstash instance does not.

To collect and ship monitoring data:

  1. Disable default collection of monitoring metrics
  2. Specify the target cluster_uuid (optional)
  3. Install and configure Metricbeat to collect monitoring data

Disable default collection of Logstash monitoring metrics

edit

The monitoring setting is in the Logstash configuration file (logstash.yml), but is commented out:

monitoring.enabled: false

Remove the # at the beginning of the line to enable the setting.

Define cluster_uuid (Optional)

edit

To bind the metrics of Logstash to a specific cluster, optionally define the monitoring.cluster_uuid in the configuration file (logstash.yml):

monitoring.cluster_uuid: PRODUCTION_ES_CLUSTER_UUID

Install and configure Metricbeat

edit
  1. Install Metricbeat on the same server as Logstash.

  2. Enable the logstash-xpack module in Metricbeat.

    To enable the default configuration in the Metricbeat modules.d directory, run:

    deb or rpm:

    metricbeat modules enable logstash-xpack

    linux or mac:

    ./metricbeat modules enable logstash-xpack

    win:

    PS > .\metricbeat.exe modules enable logstash-xpack

    For more information, see Specify which modules to run and beat module.

  3. Configure the logstash-xpack module in Metricbeat.

    The modules.d/logstash-xpack.yml file contains these settings:

      - module: logstash
    metricsets:
      - node
      - node_stats
    period: 10s
    hosts: ["localhost:9600"]
    #username: "user"
    #password: "secret"
    xpack.enabled: true

    Set the hosts, username, and password to authenticate with Logstash. For other module settings, it’s recommended that you accept the defaults.

    By default, the module collects Logstash monitoring data from localhost:9600.

    To monitor multiple Logstash instances, specify a list of hosts, for example:

    hosts: ["http://localhost:9601","http://localhost:9602","http://localhost:9603"]

    Elastic security. If the Elastic security features are enabled, provide a user ID and password so that Metricbeat can collect metrics successfully:

    1. Create a user on the production cluster that has the remote_monitoring_collector built-in role.
    2. Add the username and password settings to the module configuration file (logstash-xpack.yml).

  4. Optional: Disable the system module in the Metricbeat.

    By default, the system module is enabled. The information it collects, however, is not shown on the Stack Monitoring page in Kibana. Unless you want to use that information for other purposes, run the following command:

    metricbeat modules disable system

  5. Identify where to send the monitoring data.

    In production environments, we strongly recommend using a separate cluster (referred to as the monitoring cluster) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster.

    For example, specify the Elasticsearch output information in the Metricbeat configuration file (metricbeat.yml):

    output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] 

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

    In this example, the data is stored on a monitoring cluster with nodes es-mon-1 and es-mon-2.

    If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a hosts setting like https://es-mon-1:9200.

    The Elasticsearch monitoring features use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node.

    If the Elasticsearch security features are enabled on the monitoring cluster, you must provide a valid user ID and password so that Metricbeat can send metrics successfully:

    1. Create a user on the monitoring cluster that has the remote_monitoring_agent built-in role. Alternatively, use the remote_monitoring_user built-in user.

      If you’re using index lifecycle management, the remote monitoring user requires additional privileges to create and read indices. For more information, see <<feature-roles>>.

    2. Add the username and password settings to the Elasticsearch output information in the Metricbeat configuration file.

    For more information about these configuration options, see Configure the Elasticsearch output.

  6. Start Metricbeat to begin collecting monitoring data.
  7. View the monitoring data in Kibana.

Your monitoring setup is complete.

« Monitoring Logstash Collect Logstash monitoring data using legacy collectors »