IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

› ›

ECS in Logstash

edit

ECS in Logstash

edit

The Elastic Common Schema (ECS) is an open source specification, developed with support from the Elastic user community. ECS defines a common set of fields to be used for storing event data, such as logs and metrics, in Elasticsearch. With ECS, users can normalize event data to better analyze, visualize, and correlate the data represented in their events.

ECS compatibility

edit

Many plugins implement an ECS-compatibility mode, which causes them to produce and manipulate events in a manner that is compatible with the Elastic Common Schema (ECS).

Any plugin that supports this mode will also have an ecs_compatibility option, which allows you to configure which mode the individual plugin instance should operate in. If left unspecified for an individual plugin, the pipeline’s pipeline.ecs_compatibility setting will be observed. This allows you to configure plugins to use ECS — or to lock in your existing non-ECS behavior — in advance of your Logstash 8.0 upgrade where ECS will be enabled by default.

ECS Compatibility modes do not prevent you from explicitly configuring a plugin in a manner that conflicts with ECS. Instead, they ensure that implicit configuration avoids conflicts.

Until Logstash 8.0 and the final 7.x are released, any value for pipeline.ecs_compatibility other than disabled are considered BETA and unsupported. As we continue to release plugins with ECS Compatibility modes, having this flag set will cause even patch-level upgrades to automatically consume breaking changes in the upgraded plugins, changing the shape of data the plugin produces.

ECS v8 will be the default in the GA release of Logstash 8.0.0, and will be available at or before the final minor release of Logstash 7. We expect the scope of breaking changes in ECS 8 to be limited. We are tracking progress toward ECS v8 in a GitHub issue.

Configuring ECS

edit

ECS will be on by default in Logstash 8, but you can begin using it now by configuring individual plugins with ecs_compatibility. You can also "lock in" the existing non-ECS behavior for an entire pipeline to ensure its behavior doesn’t change when you perform your next major upgrade.

Specific plugin instance

edit

Use a plugin’s ecs_compatibility option to override the default value on the plugin instance.

For example, if you want a specific instance of the GeoIP Filter to behave without ECS compatibility, you can adjust its definition in your pipeline without affecting any other plugin instances.

filter {
  geoip {
    source => "[host][ip]"
    ecs_compatibility => disabled
  }
}

Alternatively, if you had a UDP input with a CEF codec, and wanted both to use an ECS mode while still running Logstash 7, you can adjust their definitions to specify the major version of ECS to use.

input {
  udp {
    port => 1234
    ecs_compatibility => v1
    codec => cef {
      ecs_compatibility => v1
    }
  }
}

All plugins in a given pipeline

edit

If you wish to provide a specific default value for ecs_compatibility to all plugins in a pipeline, you can do so with the pipeline.ecs_compatibility setting in your pipeline definition in config/pipelines.yml or Central Management. This setting will be used unless overridden by a specific plugin instance. If unspecified for an individual pipeline, the global value will be used.

- pipeline.id: my-legacy-pipeline
  path.config: "/etc/path/to/legacy-pipeline.config"
  pipeline.ecs_compatibility: disabled
- pipeline.id: my-ecs-pipeline
  path.config: "/etc/path/to/ecs-pipeline.config"
  pipeline.ecs_compatibility: v1

Until the final minor release of Logstash 7 that coincides with the General Availability of Logstash 8.0.0, any value for pipeline.ecs_compatibility other than disabled is considered BETA and unsupported because it will produce undesireable consequences when performing upgrades. As we continue to release updated plugins with ECS-Compatibility modes, opting into them at a pipeline or process level will cause the affected plugins to silently and automatically consume breaking changes with each upgrade, which may change the shape of data your pipeline produces.

All plugins in all pipelines

edit

Similarly, you can set the default value for the whole Logstash process by setting the pipeline.ecs_compatibility value in config/logstash.yml.

pipeline.ecs_compatibility: disabled

« Execution Model Processing Details »