collate
editcollate
editThis is a community-maintained plugin! It does not ship with Logstash by default, but it is easy to install by running bin/logstash-plugin install logstash-filter-collate
.
Collate events by time or count.
The original goal of this filter was to merge the logs from different sources by the time of log, for example, in real-time log collection, logs can be collated by amount of 3000 logs or can be collated in 30 seconds.
The config looks like this:
filter { collate { size => 3000 interval => "30s" order => "ascending" } }
Synopsis
editThis plugin supports the following configuration options:
Required configuration options:
collate { }
Available configuration options:
Details
edit
add_field
edit- Value type is hash
-
Default value is
{}
If this filter is successful, add any arbitrary fields to this event.
Field names can be dynamic and include parts of the event using the %{field}
.
Example:
filter { collate { add_field => { "foo_%{somefield}" => "Hello world, from %{host}" } } } [source,ruby] # You can also add multiple fields at once: filter { collate { add_field => { "foo_%{somefield}" => "Hello world, from %{host}" "new_field" => "new_static_value" } } }
If the event has field "somefield" == "hello"
this filter, on success,
would add field foo_hello
if it is present, with the
value above and the %{host}
piece replaced with that value from the
event. The second example would also add a hardcoded field.
add_tag
edit- Value type is array
-
Default value is
[]
If this filter is successful, add arbitrary tags to the event.
Tags can be dynamic and include parts of the event using the %{field}
syntax.
Example:
filter { collate { add_tag => [ "foo_%{somefield}" ] } } [source,ruby] # You can also add multiple tags at once: filter { collate { add_tag => [ "foo_%{somefield}", "taggedy_tag"] } }
If the event has field "somefield" == "hello"
this filter, on success,
would add a tag foo_hello
(and the second example would of course add a taggedy_tag
tag).
interval
edit- Value type is string
-
Default value is
"1m"
The interval
is the time window which how long the logs should be collated. (default 1m
)
order
edit-
Value can be any of:
ascending
,descending
-
Default value is
"ascending"
The order
collated events should appear in.
periodic_flush
edit- Value type is boolean
-
Default value is
false
Call the filter flush method at regular interval. Optional.
remove_field
edit- Value type is array
-
Default value is
[]
If this filter is successful, remove arbitrary fields from this event. Fields names can be dynamic and include parts of the event using the %{field} Example:
filter { collate { remove_field => [ "foo_%{somefield}" ] } } [source,ruby] # You can also remove multiple fields at once: filter { collate { remove_field => [ "foo_%{somefield}", "my_extraneous_field" ] } }
If the event has field "somefield" == "hello"
this filter, on success,
would remove the field with name foo_hello
if it is present. The second
example would remove an additional, non-dynamic field.
remove_tag
edit- Value type is array
-
Default value is
[]
If this filter is successful, remove arbitrary tags from the event.
Tags can be dynamic and include parts of the event using the %{field}
syntax.
Example:
filter { collate { remove_tag => [ "foo_%{somefield}" ] } } [source,ruby] # You can also remove multiple tags at once: filter { collate { remove_tag => [ "foo_%{somefield}", "sad_unwanted_tag"] } }
If the event has field "somefield" == "hello"
this filter, on success,
would remove the tag foo_hello
if it is present. The second example
would remove a sad, unwanted tag as well.