metrics

edit

The metrics filter is useful for aggregating metrics.

For example, if you have a field response that is a http response code, and you want to count each kind of response, you can do this:

    filter {
      metrics {
        meter => [ "http.%{response}" ]
        add_tag => "metric"
      }
    }

Metrics are flushed every 5 seconds by default or according to flush_interval. Metrics appear as new events in the event stream and go through any filters that occur after as well as outputs.

In general, you will want to add a tag to your metrics and have an output explicitly look for that tag.

The event that is flushed will include every meter and timer metric in the following way:

meter values

edit

For a meter => "something" you will receive the following fields:

  • "thing.count" - the total count of events
  • "thing.rate_1m" - the 1-minute rate (sliding)
  • "thing.rate_5m" - the 5-minute rate (sliding)
  • "thing.rate_15m" - the 15-minute rate (sliding)

timer values

edit

For a timer => [ "thing", "%{duration}" ] you will receive the following fields:

  • "thing.count" - the total count of events
  • "thing.rate_1m" - the 1-minute rate of events (sliding)
  • "thing.rate_5m" - the 5-minute rate of events (sliding)
  • "thing.rate_15m" - the 15-minute rate of events (sliding)
  • "thing.min" - the minimum value seen for this metric
  • "thing.max" - the maximum value seen for this metric
  • "thing.stddev" - the standard deviation for this metric
  • "thing.mean" - the mean for this metric
  • "thing.pXX" - the XXth percentile for this metric (see percentiles)

Example: computing event rate

edit

For a simple example, let’s track how many events per second are running through logstash:

    input {
      generator {
        type => "generated"
      }
    }
filter {
  if [type] == "generated" {
    metrics {
      meter => "events"
      add_tag => "metric"
    }
  }
}
output {
  # only emit events with the 'metric' tag
  if "metric" in [tags] {
    stdout {
      codec => line {
        format => "rate: %{events.rate_1m}"
      }
    }
  }
}

Running the above:

    % bin/logstash -f example.conf
    rate: 23721.983566819246
    rate: 24811.395722536377
    rate: 25875.892745934525
    rate: 26836.42375967113

We see the output includes our events 1-minute rate.

In the real world, you would emit this to graphite or another metrics store, like so:

    output {
      graphite {
        metrics => [ "events.rate_1m", "%{events.rate_1m}" ]
      }
    }

 

Synopsis

edit

This plugin supports the following configuration options:

Required configuration options:

metrics {
}

Available configuration options:

Setting Input type Required Default value

add_field

hash

No

{}

add_tag

array

No

[]

clear_interval

number

No

-1

flush_interval

number

No

5

ignore_older_than

number

No

0

meter

array

No

[]

percentiles

array

No

[1, 5, 10, 90, 95, 99, 100]

periodic_flush

boolean

No

false

rates

array

No

[1, 5, 15]

remove_field

array

No

[]

remove_tag

array

No

[]

timer

hash

No

{}

Details

edit

 

add_field

edit
  • Value type is hash
  • Default value is {}

If this filter is successful, add any arbitrary fields to this event. Field names can be dynamic and include parts of the event using the %{field}.

Example:

    filter {
      metrics {
        add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
      }
    }
[source,ruby]
    # You can also add multiple fields at once:
    filter {
      metrics {
        add_field => {
          "foo_%{somefield}" => "Hello world, from %{host}"
          "new_field" => "new_static_value"
        }
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would add field foo_hello if it is present, with the value above and the %{host} piece replaced with that value from the event. The second example would also add a hardcoded field.

add_tag

edit
  • Value type is array
  • Default value is []

If this filter is successful, add arbitrary tags to the event. Tags can be dynamic and include parts of the event using the %{field} syntax.

Example:

    filter {
      metrics {
        add_tag => [ "foo_%{somefield}" ]
      }
    }
[source,ruby]
    # You can also add multiple tags at once:
    filter {
      metrics {
        add_tag => [ "foo_%{somefield}", "taggedy_tag"]
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would add a tag foo_hello (and the second example would of course add a taggedy_tag tag).

clear_interval

edit
  • Value type is number
  • Default value is -1

The clear interval, when all counter are reset.

If set to -1, the default value, the metrics will never be cleared. Otherwise, should be a multiple of 5s.

exclude_tags (DEPRECATED)

edit
  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value type is array
  • Default value is []

Only handle events without any of these tags. Optional.

flush_interval

edit
  • Value type is number
  • Default value is 5

The flush interval, when the metrics event is created. Must be a multiple of 5s.

ignore_older_than

edit
  • Value type is number
  • Default value is 0

Don’t track events that have @timestamp older than some number of seconds.

This is useful if you want to only include events that are near real-time in your metrics.

Example, to only count events that are within 10 seconds of real-time, you would do this:

filter {
  metrics {
    meter => [ "hits" ]
    ignore_older_than => 10
  }
}

meter

edit
  • Value type is array
  • Default value is []

syntax: meter => [ "name of metric", "name of metric" ]

percentiles

edit
  • Value type is array
  • Default value is [1, 5, 10, 90, 95, 99, 100]

The percentiles that should be measured

periodic_flush

edit
  • Value type is boolean
  • Default value is false

Call the filter flush method at regular interval. Optional.

rates

edit
  • Value type is array
  • Default value is [1, 5, 15]

The rates that should be measured, in minutes. Possible values are 1, 5, and 15.

remove_field

edit
  • Value type is array
  • Default value is []

If this filter is successful, remove arbitrary fields from this event. Fields names can be dynamic and include parts of the event using the %{field} Example:

    filter {
      metrics {
        remove_field => [ "foo_%{somefield}" ]
      }
    }
[source,ruby]
    # You can also remove multiple fields at once:
    filter {
      metrics {
        remove_field => [ "foo_%{somefield}", "my_extraneous_field" ]
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would remove the field with name foo_hello if it is present. The second example would remove an additional, non-dynamic field.

remove_tag

edit
  • Value type is array
  • Default value is []

If this filter is successful, remove arbitrary tags from the event. Tags can be dynamic and include parts of the event using the %{field} syntax.

Example:

    filter {
      metrics {
        remove_tag => [ "foo_%{somefield}" ]
      }
    }
[source,ruby]
    # You can also remove multiple tags at once:
    filter {
      metrics {
        remove_tag => [ "foo_%{somefield}", "sad_unwanted_tag"]
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would remove the tag foo_hello if it is present. The second example would remove a sad, unwanted tag as well.

tags (DEPRECATED)

edit
  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value type is array
  • Default value is []

Only handle events with all of these tags. Optional.

timer

edit
  • Value type is hash
  • Default value is {}

syntax: timer => [ "name of metric", "%{time_value}" ]

type (DEPRECATED)

edit
  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value type is string
  • Default value is ""

Note that all of the specified routing options (type,tags,exclude_tags,include_fields, exclude_fields) must be met in order for the event to be handled by the filter. The type to act on. If a type is given, then this filter will only act on messages with the same type. See any input plugin’s type attribute for more. Optional.