Writing Metricbeat modules based on other modules: Introducing light modules

By
Jaime Soriano

Metricbeat has modules that support the collection of metrics using specific protocols in a generic way, allowing you to leverage the Elastic Stack for fetching, storing, and analyzing information coming from other existing solutions. These generic modules also provide a way of extending Metricbeat functionality by monitoring services that don’t have their own native modules. Some examples are the modules for Prometheus, Jolokia, and AWS cloudwatch.

Internally, we sometimes call these modules “input” modules, because of the parallelism we find between them and Filebeat inputs. Filebeat inputs can be used on their own, but they are also the foundations of Filebeat modules. Filebeat modules can be seen as packaged Filebeat configurations that cover specific services or use cases. They work by collecting information using inputs, but also include documentation, field mappings, and dashboards, extending filebeat functionality without Go code. We wondered if we could have something similar in Metricbeat, and thus, we created the concept of Metricbeat light modules.

Light modules are a new way of defining Metricbeat modules based on existing implementations. They don't contain any Go code — all functionality is provided by existing metricsets. They can be seen as predefined configurations for generic metricsets. They speed up the development of new modules for services whose monitoring is based on protocols we already support, and they provide a way to share existing Metricbeat configurations and dashboards.

Implementing Metricbeat light modules

Metricbeat light modules keep the known structure of modules and metricsets, but instead of using Go code to define the modules and metricsets, files containing definitions as metadata are used.

Each light module includes a module.yml file with the list of light metricsets, and for each metricset, there is a directory with a manifest.yml with its definition. Light modules included in the beats repository can also provide documentation files, dashboards, field mappings, or automated tests, just like any other module.

A basic light module with two metricsets would have a structure like this one: 

├── module.yml
├── _meta
│   ├── config.yml
│   └── docs.asciidoc
├── metricset1
│   ├── manifest.yml
│   └── _meta
│       └── docs.asciidoc
└── metricset2
    ├── manifest.yml
    └── _meta
        └── docs.asciidoc

A manifest file includes a reference to an existing metricset, allowing to override or define new defaults. For example the following file would define a module based on the metricset somemetricset of the module somemodule, and would set a couple of default settings: 

input:
  module: somemodule
  metricset: somemetricset
  defaults:
    some_option: foo
    other_option: bar
  default: true

A Metricbeat user could then use it with a configuration similar to the one of any other module: 

- module: module
  metricsets: ['metricset1', 'metricset2']
  ...

Light modules are designed as a fallback mechanism to current modules — if a module is included in Metricbeat configuration and is not found along the registered ones, then Metricbeat looks for an existing light module and metricset with the given name in its home path. If it is found, it is loaded and instantiated.

As we have seen, a light module manifest references a metricset as its input implementation. The reference metricset must be an existing native metricset — it cannot reference another light module. The final module would be instantiated and customized with the default settings defined in the manifest.

There can be modules that mix native metricsets with light metricsets, for example we plan to implement some AWS metricsets, like elb, as light modules based on the AWS cloudwatch metricset.

The first module implemented as a light module is for CockroachDB; its beta version is included in Metricbeat 7.3.

Example: CockroachDB light module

CockroachDB is a good service to use as an initial example because it can be monitored using an existing metricset — the Prometheus collector — but it requires a small customization to set the proper metrics endpoint (it uses /_status/vars as the metrics path instead of the default /metrics). If we wanted to monitor CockroachDB with the Prometheus collector metricset, we could do it using a configuration like the following one: 

- module: prometheus
  metricsets: ['collector']
  hosts: ['localhost:8080']
  metrics_path: '/_status/vars'

If we want to migrate this configuration to a light module, we can do it with the following files:

x-pack/metricbeat/module/cockroachdb/module.yml

name: cockroachdb
metricsets:
- status

x-pack/metricbeat/module/cockroachdb/status/manifest.yml

input:
  module: prometheus
  metricset: collector
  defaults:
    metrics_path: /_status/vars
default: true

With this module definition we are telling Metricbeat to use the collector metricset of the prometheus module, but replacing the default metrics_path with /_status/vars. We are also indicating that this is a default metricset of this module, so it is instantiated if no other metricset is specified in the configuration file.

Despite not having code, testing of this module is done in a similar way to other modules. But because its definition won't be included in the test binary files, we have to take a few things into account.

First, base module and metricset may not be registered in the modules registry of the testing binary, so we have to explicitly import it so it gets registered: 

import (
  _ "github.com/elastic/beats/metricbeat/module/prometheus"
  _ "github.com/elastic/beats/metricbeat/module/prometheus/collector"
)

And the modules registry doesn’t know anything about light modules in test binaries, so we have to explicitly indicate where they are located. They will be loaded from this location as configuration files, so we also remove the permission checks that are applied to these files. Both things can be done in an init() function in the test files of the module: 

func init() {
  os.Setenv("BEAT_STRICT_PERMS", "false")
  mb.Registry.SetSecondarySource(
    xpackmb.NewLightModulesSource("../../../module"))
}

Once the module is ready, it can be used in a way that feels familiar to Metricbeat users: 

- module: cockroachdb
  metricsets: ['status']
  hosts: ['localhost:8080']

Events generated look like these: 

{
 "@timestamp": "2019-05-22T10:00:30.818Z",
 "event": {
  "dataset": "cockroachdb.status",
  "duration": 29065910,
  "module": "cockroachdb"
 },
 "metricset": {
  "name": "status"
 },
 "prometheus": {
  "labels": {
   "le": "1638399",
   "store": "1"
  },
  "metrics": {
   "raft_process_applycommitted_latency_bucket": 106946,
   "raft_process_commandcommit_latency_bucket": 111112,
   "raft_process_handleready_latency_bucket": 21243,
   "raft_process_logcommit_latency_bucket": 53080
  }
 },
 "service": {
  "address": "172.17.0.2:8080",
  "type": "cockroachdb"
 }
}

They are events like the ones that would be generated by the Prometheus collector metricset if used on its own, but with the event and service information replaced to indicate the module and dataset used.

Prometheus namespacing is intentionally kept so it reuses the fields mappings of the original metricset.

Finally, this module also includes a dashboard that makes use of some of the available metrics, like:

  • prometheus.metrics.sql_conns for the number of active SQL connections
  • prometheus.metrics.replicas for the number of replicas
  • prometheus.metrics.ranges_unavailable for the number of unavailable ranges

Dashboard showing active SQL connections, replicas, and unavailable ranges

What's next?

There are many use cases where this new framework for developing Metricbeat modules can be useful. Besides the example based on Prometheus seen in this article, we are already working on the AWS ELB metricset based on the cloudwatch metricset, and we also plan to start working on supporting more Java services using the Jolokia module.

On the tooling side, we think that metricbeat subcommands give users a complete toolset for contributing new modules without more requirements than the Elastic Stack itself. In this sense we are thinking about ways to extend metricbeat test, or introduce new subcommands that can be used for light modules validation.

Finally, we invite you to create your own light modules. If you're already using one of the mentioned generic modules to monitor some services, consider migrating this configuration to a module others can use.

If you want to learn more, take a look at the design issue, the initial implementation, and the pull request for the CockroachDB module. And if you have any questions, remember that we are always happy to help on the Discuss forums.