Customers

Dockbeat: A new addition to the Beats Community

By
Douae Jeouit
•
Thibaut Marmin
•
Erwann Cloarec

Disclaimer: The previously published article has now been modified to reflect the name change from DockerBeat to Dockbeat.

Did you ever want to know how your Docker containers behave over time? Did you ever want to have a Beat capable of reading Docker containers statistics and indexing them into Elasticsearch? Today, the solution is at your fingertips: Dockbeat! 

Dockbeat is an open source project created by Ingensi, based on the Beats platform. Ingensi is a business unit of Cyres Group located in France. Our main goal is to create big data solutions offering powerful decision-making tools to unveil the value in the customer data. For that, we employ our technological expertise in Hadoop, Elasticsearch, and Docker.

At Ingensi, we use the Elastic Stack as the basis of our log management solution. We also use Elasticsearch as a search engine in our application as we need fast responding searches.

Docker Big Data platform: Ingensi meets Docker

For resource optimization reasons, when we receive a cluster deployment request from our customers, we deploy it in a shared big data infrastructure based on Docker. In 2015, we had our first shared big data infrastructure deployed and we naturally used the Elastic Stack to monitor the entire platform: hosts, network, users, etc. However, our processing workflow did not allow us to monitor Docker containers.

That’s when Beats comes into play! The Beats platform allows you to build lightweight data shippers for different types of data that you can search and analyze in Elasticsearch, and visualize in Kibana. From there, we had the idea to create our own beat to collect Docker statistics.

Hello, Dockbeat!

Since the release of Docker 1.5., a new functionality was added to Docker's API: Docker stats. This endpoint returns a live stream of a running container's resource usage statistics, such as CPU, memory, network, disk IO, etc.

Therefore, we've decided to collect all these metrics in order to capitalize on the use of our existing Docker infrastructure. Based on the Beats platform, we started to develop our own Beat: Dockbeat.

How does Dockbeat work?

Dockbeat is used for Docker monitoring. It is responsible for collecting the containers' statistics and send them to Logstash or directly to Elasticsearch. As it is very lightweight, Dockbeat’s impct on the server is very low. The collected data can then be visualized in a Kibana dashboard. Today, we are able to view in real time all the containers running on our servers, their consumption, and thus, quickly identify any anomalies. This new beat exports five types of documents corresponding to the different metric sets collected :

  • Container: container’s attributes
  • CPU: container's CPU usage statistics
  • Net: container's network statistics
  • Memory: container's memory statistics
  • Blkio: container’s IO access statistics

dockbeat-workflow.png

Figure 1 - Dockbeat Workflow

Great! How can I use Dockbeat?

Simple: Dockbeat is hosted on GitHub. You first need to clone and build the project (see README) or simply download the latest release version you need.

In a basic Docker installation, Dockbeat should work out of the box with its default configuration. If you have your own settings, check the configuration file dockbeat.yml. In this file you can configure: 

  • The Docker socket path
  • The output that receives the collected data by the beat. Multiple outputs may be used: Elasticsearch, Logstash, Kafka as well as files
  • The rate at which Docker stats are sent to the defined output (basically Elasticsearch)
  • Whether the Dockbeat has to deal with TLS docker daemon. If so, then you need to enable TLS and configure path for key and certificates.

There is no specific configuration to be edited, unless you need to enable the TLS docker daemon.

    Finally, you can launch your beat by running the following command : 

    ./dockbeat -c etc/dockbeat.yml

    dockerbeat-kibana-dashboard.png

    Figure 2 - Dockbeat Kibana Dashboard

    Dockbeat today

    Dockbeat grew up and became operational. Today, we are happy to announce the release version 1.0.0 It's an open source project and we love to receive contributions from the community. You can contribute to Dockbeat in many ways: 

    • Write documentation, tutorials, blog posts 
    • Submit bug reports or ideas for new features via GitHub 
    • Implement new features, etc. 

    So, feel free to contact us for further information or contributions, we will be happy to hear from you.

    The Team

    Erwann Cloarec is a Big Data Engineer @Ingensi, Docker passionate, Elastic fan and Hadoop professional, Dockbeat co-creator.
    @ErwannC Github LinkedIn

    Thibaut Marmin is a Software Engineer @Ingensi. Interested in Big Data and OpenSource technologies, Docker & Elastic enthusiast, Dockbeat co-creator.
    @thibmarmin Github LinkedIn

    Douae Jeouit is an Engineering Student in final year. Curious and ambitious, she loves to discover new intelligent technologies. Elastic fan. She's the blog post author and a Dockbeat contributor.
    Github LinkedIn