Creating a Metricset

A metricset is the part of a Metricbeat module that fetches and structures the data from the remote service. Each module can have multiple metricsets. In this guide, you learn how to create your own metricset. If you want to create your own Beat that uses Metricbeat as a library, see Creating a Beat based on Metricbeat.

When creating a metricset for the first time, it generally helps to look at the implementation of existing metricsets for inspiration.

To create a new metricset:

After running the make commands, you’ll find the metricset, along with its generated files, under module/{module}/{metricset}. This directory contains the following files:

Let’s look at the files in more detail next.

The first file is {metricset}.go. It contains the logic on how to fetch data from the service and convert it for sending to the output.

The generated file looks like this:

https://github.com/elastic/beats/blob/master/metricbeat/scripts/module/metricset/metricset.go.tmpl

package {metricset}

import (
	"github.com/elastic/beats/libbeat/common"
	"github.com/elastic/beats/libbeat/logp"
	"github.com/elastic/beats/metricbeat/mb"
)

// init registers the MetricSet with the central registry.
// The New method will be called after the setup of the module and before starting to fetch data
func init() {
	if err := mb.Registry.AddMetricSet("{module}", "{metricset}", New); err != nil {
		panic(err)
	}
}

// MetricSet type defines all fields of the MetricSet
// As a minimum it must inherit the mb.BaseMetricSet fields, but can be extended with
// additional entries. These variables can be used to persist data or configuration between
// multiple fetch calls.
type MetricSet struct {
	mb.BaseMetricSet
	counter int
}

// New create a new instance of the MetricSet
// Part of new is also setting up the configuration by processing additional
// configuration entries if needed.
func New(base mb.BaseMetricSet) (mb.MetricSet, error) {

	config := struct{}{}

	logp.Warn("EXPERIMENTAL: The {module} {metricset} metricset is experimental")

	if err := base.Module().UnpackConfig(&config); err != nil {
		return nil, err
	}

	return &MetricSet{
		BaseMetricSet: base,
		counter:       1,
	}, nil
}

// Fetch methods implements the data gathering and data conversion to the right format
// It returns the event which is then forward to the output. In case of an error, a
// descriptive error must be returned.
func (m *MetricSet) Fetch() (common.MapStr, error) {

	event := common.MapStr{
		"counter": m.counter,
	}
	m.counter++

	return event, nil
}

The package clause and import declaration are part of the base structure of each Golang file. You should only modify this part of the file if your implementation requires more imports.

The init method registers the metricset with the central registry. In Golang the init() function is called before the execution of all other code. This means the module will be automatically registered with the global registry.

The New method, which is passed to AddMetricSet, will be called after the setup of the module and before starting to fetch data. You normally don’t need to change this part of the file.

func init() {
	if err := mb.Registry.AddMetricSet("{module}", "{metricset}", New); err != nil {
		panic(err)
	}
}

The MetricSet type defines all fields of the metricset. As a minimum it must inherit the mb.BaseMetricSet fields, but can be extended with additional entries. These variables can be used to persist data or configuration between multiple fetch calls.

You can add more fields to the MetricSet type, as you can see in the following example where the username and password string fields are added:

type MetricSet struct {
	mb.BaseMetricSet
	username    string
	password    string
}

The New function creates a new instance of the MetricSet. The setup process of the MetricSet is also part of New. This method will be called before Fetch is called the first time.

The New function also sets up the configuration by processing additional configuration entries, if needed.

func New(base mb.BaseMetricSet) (mb.MetricSet, error) {

	config := struct{}{}

	if err := base.Module().UnpackConfig(&config); err != nil {
		return nil, err
	}

	return &MetricSet{
		BaseMetricSet: base,
	}, nil
}

The Fetch method is the central part of the metricset. Fetch is called every time new data is retrieved. If more than one host is defined, Fetch is called once for each host. The frequency of calling Fetch is based on the period defined in the configuration file.

Fetch must return a common.MapStr object, which is then sent to Elasticsearch. If an error happens, the error must be returned and then is sent instead to Elasticsearch. This means that Metricbeat always sends an event, even on failure. You must make sure that the error message helps to identify the actual error.

The following example shows a metricset Fetch method with a counter that is incremented for each Fetch call:

func (m *MetricSet) Fetch() (common.MapStr, error) {

	event := common.MapStr{
		"counter": m.counter,
	}
	m.counter++

	return event, nil
}

Fetch must return a common.MapStr, which will be translated to the JSON content. The JSON output will be identical to the naming and structure you use in common.MapStr. For more details about MapStr and its functions, see the MapStr API docs.

Metricbeat has two different Fetch interfaces. One of the interfaces, which you saw in the previous example, fetches a single event. In some cases, every fetch returns a list of events. Instead of using an array inside a JSON document, we recommend that you create a list of events.

For this kind of data, you can use the following Fetch interface:

(m *MetricSet) Fetch() ([]common.MapStr, error)

The only difference between this and the previous example is that the second example returns []common.MapStr. Metricbeat will add the same timestamp to all the events in the list to make it possible to correlate the events.

In Metricbeat we aim to normalize the metric names from all metricsets to respect a common set of conventions. This makes it easy for users to find and interpret metrics. To simplify parsing, converting, renaming, and restructuring of the object read from the monitored system to the Metricbeat format, we have created the schema package that allows you to declaratively define transformations.

For example, assuming this input object:

input := map[string]interface{}{
	"testString":     "hello",
	"testInt":        "42",
	"testBool":       "true",
	"testFloat":      "42.1",
	"testObjString":  "hello, object",
}

And the requirement to transform it into this one:

common.MapStr{
	"test_string": "hello",
	"test_int":    int64(42),
	"test_bool":   true,
	"test_float":  42.1,
	"test_obj": common.MapStr{
		"test_obj_string": "hello, object",
	},
}

You can use the following code to make the transformations:

import (
	s "github.com/elastic/beats/libbeat/common/schema"
	c "github.com/elastic/beats/libbeat/common/schema/mapstrstr"
)

var (
	schema = s.Schema{
		"test_string": c.Str("testString"),
		"test_int":    c.Int("testInt"),
		"test_bool":   c.Bool("testBool"),
		"test_float":  c.Float("testFloat"),
		"test_obj": s.Object{
			"test_obj_string": c.Str("testObjString"),
		},
	}
)

func eventMapping(input map[string]interface{}) common.MapStr {
	return schema.Apply(input)
}

In the above example, note that it is possible to create the schema object once and apply it to all events.

The configuration file for a metricset is handled by the module. If there are multiple metricsets in one module, make sure you add all metricsets to the configuration. For example:

metricbeat:
  modules:
    - module: {module-name}
      metricsets: ["{metricset1}", "{metricset2}"]

For more details about the Metricbeat configuration file, see the topic about Modules in the Metricbeat documentation.

This topic provides basic steps for creating a metricset. For more details about metricsets and how to extend your metricset further, see Metricset Details.