New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Settings for internal collection Use legacy internal collection to send monitoring data »
Elastic Docs ›Filebeat Reference [7.4] ›Monitoring Filebeat

Use Metricbeat to send monitoring data

edit

Use Metricbeat to send monitoring data

edit

In 7.3 and later, you can use Metricbeat to collect data about Filebeat and ship it to the monitoring cluster. The benefit of using Metricbeat instead of internal collection is that the monitoring agent remains active even if the Filebeat instance dies.

To learn about monitoring in general, see Monitor a cluster.

To collect and ship monitoring data:

  1. Configure the shipper you want to monitor
  2. Install and configure Metricbeat to collect monitoring data

Configure the shipper you want to monitor

edit

  1. Enable the HTTP endpoint to allow external collection of monitoring data:

    Add the following setting in the Filebeat configuration file (filebeat.yml):

    http.enabled: true

    By default, metrics are exposed on port 5066. If you need to monitor multiple Beats shippers running on the same server, set http.port to expose metrics for each shipper on a different port number:

    http.port: 5067

  2. Disable the default collection of Filebeat monitoring metrics.

    Add the following setting in the Filebeat configuration file (filebeat.yml):

    monitoring.enabled: false

    For more information, see Monitoring configuration options.

  3. Start Filebeat.

Install and configure Metricbeat to collect monitoring data

edit
  1. Install Metricbeat on the same server as Filebeat. If you already have Metricbeat installed on the server, skip this step.

  2. Enable the beat-xpack module in Metricbeat.

    For example, to enable the default configuration in the modules.d directory, run the following command, using the correct command syntax for your OS:

    metricbeat modules enable beat-xpack

    For more information, see Specify which modules to run and beat module.

  3. Configure the beat-xpack module in Metricbeat.

    The modules.d/beat-xpack.yml file contains the following settings:

    - module: beat
  metricsets:
    - stats
    - state
  period: 10s
  hosts: ["http://localhost:5066"]
  #username: "user"
  #password: "secret"
  xpack.enabled: true

    Set the hosts, username, and password settings as required by your environment. For other module settings, it’s recommended that you accept the defaults.

    By default, the module collects Filebeat monitoring data from localhost:5066. If you exposed the metrics on a different host or port when you enabled the HTTP endpoint, update the hosts setting.

    To monitor multiple Beats agents, specify a list of hosts, for example:

    hosts: ["http://localhost:5066","http://localhost:5067","http://localhost:5068"]

    If you configured Filebeat to use encrypted communications, you must access it via HTTPS. For example, use a hosts setting like https://localhost:5066.

    If the Elastic security features are enabled, you must also provide a user ID and password so that Metricbeat can collect metrics successfully:

    1. Create a user on the production cluster that has the remote_monitoring_collector built-in role. Alternatively, if it’s available in your environment, use the remote_monitoring_user built-in user.
    2. Add the username and password settings to the beat module configuration file.

  4. Optional: Disable the system module in the Metricbeat.

    By default, the system module is enabled. The information it collects, however, is not shown on the Stack Monitoring page in Kibana. Unless you want to use that information for other purposes, run the following command:

    metricbeat modules disable system

  5. Identify where to send the monitoring data.

    In production environments, we strongly recommend using a separate cluster (referred to as the monitoring cluster) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster.

    For example, specify the Elasticsearch output information in the Metricbeat configuration file (metricbeat.yml):

    output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] 

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

    In this example, the data is stored on a monitoring cluster with nodes es-mon-1 and es-mon-2.

    If you configured the monitoring cluster to use encrypted communications, you must access it via HTTPS. For example, use a hosts setting like https://es-mon-1:9200.

    The Elasticsearch monitoring features use ingest pipelines, therefore the cluster that stores the monitoring data must have at least one ingest node.

    If the Elasticsearch security features are enabled on the monitoring cluster, you must provide a valid user ID and password so that Metricbeat can send metrics successfully:

    1. Create a user on the monitoring cluster that has the remote_monitoring_agent built-in role. Alternatively, if it’s available in your environment, use the remote_monitoring_user built-in user.

      If you’re using index lifecycle management, the remote monitoring user requires additional privileges to create and read indices. For more information, see Grant users access to secured resources.

    2. Add the username and password settings to the Elasticsearch output information in the Metricbeat configuration file.

    For more information about these configuration options, see Configure the Elasticsearch output.

  6. Start Metricbeat to begin collecting monitoring data.
  7. View the monitoring data in Kibana.
« Settings for internal collection Use legacy internal collection to send monitoring data »