Monitoring Logstash

edit

To monitor Logstash nodes:

  1. Install X-Pack by running bin/logstash-plugin install in the Logstash installation directory on each node you want to monitor:

    bin/logstash-plugin install x-pack

    The plugin install script requires direct Internet access to download and install X-Pack. If your Logstash node doesn’t have Internet access, manually download and install X-Pack.

  2. Configure your Logstash nodes to send metrics to your Elasticsearch cluster by setting the xpack.monitoring.elasticsearch.url in logstash.yml. If security is enabled, you also need to specify the credentials for the built-in logstash_system user.

    To visualize Logstash as part of the Stack (shown below in Step 5), send metrics to your production cluster. Sending metrics to a dedicated monitoring cluster will show the Logstash metrics under the monitoring cluster.

    xpack.monitoring.elasticsearch.url: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] 
    xpack.monitoring.elasticsearch.username: "logstash_system" 
    xpack.monitoring.elasticsearch.password: "changeme"

    If SSL/TLS is enabled on the production cluster, you must connect through HTTPS. As of v5.2.1, you can specify multiple Elasticsearch hosts as an array as well as specifying a single host as a string. If multiple URLs are specified, Logstash can round-robin requests to these production nodes.

    The logstash_system user is automatically added when you install X-Pack. Don’t forget to change the default passwords for all of the built in users. If X-Pack security is disabled on the production cluster, you can omit the username and password for the agent.

  3. If SSL/TLS is enabled on the production Elasticsearch cluster, specify the trusted CA certificate(s) that will be used to verify the identity of the nodes in the cluster.

    To add a CA certificate to a Logstash node’s trusted certificates, you can specify the location of the PEM encoded certificate with the ca setting:

    xpack.monitoring.elasticsearch.ssl.ca: [ "/path/to/ca.crt" ]

    Alternatively, you can configure trusted certificates using a truststore (a Java Keystore file that contains the certificates):

    xpack.monitoring.elasticsearch.ssl.truststore.path: /path/to/file
    xpack.monitoring.elasticsearch.ssl.truststore.password: password

    Also, optionally, you can set up client certificate using a keystore (a Java Keystore file that contains the certificate):

    xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
    xpack.monitoring.elasticsearch.ssl.keystore.password: password
  4. Restart your Logstash nodes.
  5. To verify your X-Pack monitoring configuration, point your web browser at your Kibana host, and select Monitoring from the side navigation. Metrics reported from your Logstash nodes should be visible in the Logstash section. When security is enabled, to view the monitoring dashboards you must log in to Kibana as a user who has the kibana_user and monitoring_user roles.

    Monitoring

Re-enabling Logstash Monitoring After Upgrading

edit

When upgrading from older versions of X-Pack, the built-in logstash_system user is disabled for security reasons. To resume monitoring, change the password and re-enable the logstash_system user.