« Repositories for APT and YUM Run Metricbeat on Kubernetes »
Elastic Docs ›Metricbeat Reference [8.16] ›Set up and run Metricbeat

Run Metricbeat on Docker

edit

Run Metricbeat on Docker

edit

Docker images for Metricbeat are available from the Elastic Docker registry. The base image is centos:7.

A list of all published Docker images and tags is available at www.docker.elastic.co.

These images are free to use under the Elastic license. They contain open source and free commercial features and access to paid commercial features. Start a 30-day trial to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

Pull the image

edit

Obtaining Metricbeat for Docker is as simple as issuing a docker pull command against the Elastic Docker registry.

docker pull docker.elastic.co/beats/metricbeat:8.16.1

Alternatively, you can download other Docker images that contain only features available under the Apache 2.0 license. To download the images, go to www.docker.elastic.co.

Optional: Verify the image

edit

You can use the Cosign application to verify the Metricbeat Docker image signature.

wget https://artifacts.elastic.co/cosign.pub
cosign verify --key cosign.pub docker.elastic.co/beats/metricbeat:8.16.1

The cosign command prints the check results and the signature payload in JSON format:

Verification for docker.elastic.co/beats/metricbeat:8.16.1 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key

Run the Metricbeat setup

edit

Running Metricbeat with the setup command will create the index pattern and load visualizations , dashboards, and machine learning jobs. Run this command:

docker run --rm \
docker.elastic.co/beats/metricbeat:8.16.1 \
setup -E setup.kibana.host=kibana:5601 \
-E output.elasticsearch.hosts=["elasticsearch:9200"]

Substitute your Kibana and Elasticsearch hosts and ports.

If you are using the hosted Elasticsearch Service in Elastic Cloud, replace the -E output.elasticsearch.hosts line with the Cloud ID and elastic password using this syntax:

 
-E cloud.id=<Cloud ID from Elasticsearch Service> \
-E cloud.auth=elastic:<elastic password>

Run Metricbeat on a read-only file system

edit

If you’d like to run Metricbeat in a Docker container on a read-only file system, you can do so by specifying the --read-only option. Metricbeat requires a stateful directory to store application data, so with the --read-only option you also need to use the --mount option to specify a path to where that data can be stored.

For example:

docker run --rm \
  --mount type=bind,source=$(pwd)/data,destination=/usr/share/metricbeat/data \
  --read-only \
  docker.elastic.co/beats/metricbeat:8.16.1

Configure Metricbeat on Docker

edit

The Docker image provides several methods for configuring Metricbeat. The conventional approach is to provide a configuration file via a volume mount, but it’s also possible to create a custom image with your configuration included.

Example configuration file

edit

Download this example configuration file as a starting point:

curl -L -O https://raw.githubusercontent.com/elastic/beats/8.16/deploy/docker/metricbeat.docker.yml

Volume-mounted configuration

edit

One way to configure Metricbeat on Docker is to provide metricbeat.docker.yml via a volume mount. With docker run, the volume mount can be specified like this.

docker run -d \
  --name=metricbeat \
  --user=root \
  --volume="$(pwd)/metricbeat.docker.yml:/usr/share/metricbeat/metricbeat.yml:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  --volume="/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro" \
  --volume="/proc:/hostfs/proc:ro" \
  --volume="/:/hostfs:ro" \
  docker.elastic.co/beats/metricbeat:8.16.1 metricbeat -e \
  -E output.elasticsearch.hosts=["elasticsearch:9200"]

Substitute your Elasticsearch hosts and ports.

If you are using the hosted Elasticsearch Service in Elastic Cloud, replace the -E output.elasticsearch.hosts line with the Cloud ID and elastic password using the syntax shown earlier.

Customize your configuration

edit

The metricbeat.docker.yml file you downloaded earlier is configured to deploy Beats modules based on the Docker labels applied to your containers. See Hints based autodiscover for more details. Add labels to your application Docker containers, and they will be picked up by the Beats autodiscover feature when they are deployed. Here is an example command for an Apache HTTP Server container with labels to configure the Filebeat and Metricbeat modules for the Apache HTTP Server:

docker run \
  --label co.elastic.logs/module=apache2 \
  --label co.elastic.logs/fileset.stdout=access \
  --label co.elastic.logs/fileset.stderr=error \
  --label co.elastic.metrics/module=apache \
  --label co.elastic.metrics/metricsets=status \
  --label co.elastic.metrics/hosts='${data.host}:${data.port}' \
  --detach=true \
  --name my-apache-app \
  -p 8080:80 \
  httpd:2.4

Custom image configuration

edit

It’s possible to embed your Metricbeat configuration in a custom image. Here is an example Dockerfile to achieve this:

FROM docker.elastic.co/beats/metricbeat:8.16.1
COPY --chown=root:metricbeat metricbeat.yml /usr/share/metricbeat/metricbeat.yml

Monitor the host machine

edit

When executing Metricbeat in a container, there are some important things to be aware of if you want to monitor the host machine or other containers. Let’s walk-through some examples using Docker as our container orchestration tool.

This example highlights the changes required to make the system module work properly inside of a container. This enables Metricbeat to monitor the host machine from within the container.

docker run \
  --mount type=bind,source=/proc,target=/hostfs/proc,readonly \ 
  --mount type=bind,source=/sys/fs/cgroup,target=/hostfs/sys/fs/cgroup,readonly \ 
  --mount type=bind,source=/,target=/hostfs,readonly \ 
  --mount type=bind,source=/var/run/dbus/system_bus_socket,target=/hostfs/var/run/dbus/system_bus_socket,readonly \ 
  --env DBUS_SYSTEM_BUS_ADDRESS='unix:path=/hostfs/var/run/dbus/system_bus_socket' \ 
  --net=host \ 
  --cgroupns=host \ 
  docker.elastic.co/beats/metricbeat:8.16.1 -e -system.hostfs=/hostfs

Metricbeat’s system module collects much of its data through the Linux proc filesystem, which is normally located at /proc. Because containers are isolated as much as possible from the host, the data inside of the container’s /proc is different than the host’s /proc. To account for this, you can mount the host’s /proc filesystem inside of the container and tell Metricbeat to look inside the /hostfs directory when looking for /proc by using the hostfs=/hostfs config value.

By default, cgroup reporting is enabled for the system process metricset, so you need to mount the host’s cgroup mountpoints within the container. They need to be mounted inside the directory specified by the hostfs config value.

If you want to be able to monitor filesystems from the host by using the system filesystem metricset, then those filesystems need to be mounted inside of the container. They can be mounted at any location.

The system users metricset and system service metricset both require access to dbus. Mount the dbus socket and set the DBUS_SYSTEM_BUS_ADDRESS environment variable to the mounted system socket path.

The system network metricset uses data from /proc/net/dev, or /hostfs/proc/net/dev when using hostfs=/hostfs. The only way to make this file contain the host’s network devices is to use the --net=host flag. This is due to Linux namespacing; simply bind mounting the host’s /proc to /hostfs/proc is not sufficient.

Runs the container using the host’s cgroup namespace, instead of a private namespace. While this is optional, system process metricset may produce more correct cgroup metrics when running in host mode.

The special filesystems /proc and /sys are only available if the host system is running Linux. Attempts to bind-mount these filesystems will fail on Windows and MacOS.

If the system socket metricset is being used on Linux, more privileges will need to be granted to Metricbeat. This metricset reads files from /proc that are an interface to internal objects owned by other users. The capabilities needed to read all these files (sys_ptrace and dac_read_search) are disabled by default on Docker. To grant these permissions these flags are needed too:

--user root --cap-add sys_ptrace --cap-add dac_read_search

Monitor a service in another container

edit

Next, let’s look at an example of monitoring a containerized service from a Metricbeat container.

docker run \
  --network=mysqlnet \ 
  -e MYSQL_PASSWORD=secret \ 
  docker.elastic.co/beats/metricbeat:8.16.1

Placing the Metricbeat and MySQL containers on the same Docker network allows Metricbeat access to the exposed ports of the MySQL container, and makes the hostname mysql resolvable to Metricbeat.

If you do not want to hardcode certain values into your Metricbeat configuration, then you can pass them into the container either as environment variables or as command line flags to Metricbeat (see the -E CLI flag in Command reference).

The mysql module configuration would look like this:

metricbeat.modules:
- module: mysql
  metricsets: ["status"]
  hosts: ["tcp(mysql:3306)/"] 
  username: root
  password: ${MYSQL_PASSWORD}

The mysql hostname will resolve to the address of a container named mysql on the mysqlnet Docker network.

The MYSQL_PASSWORD variable will be evaluated at startup. If the variable is not set, this will lead to an error at startup.
« Repositories for APT and YUM Run Metricbeat on Kubernetes »