Cef codec plugin
editCef codec plugin
edit- Plugin version: v6.2.3
- Released on: 2021-07-29
- Changelog
For other versions, see the Versioned plugin docs.
Getting Help
editFor questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.
Description
editImplementation of a Logstash codec for the ArcSight Common Event Format (CEF) Based on Revision 20 of Implementing ArcSight CEF, dated from June 05, 2013 https://community.saas.hpe.com/dcvta86296/attachments/dcvta86296/connector-documentation/1116/1/CommonEventFormatv23.pdf
If this codec receives a payload from an input that is not a valid CEF message, then it will produce an event with the payload as the message field and a _cefparsefailure tag.
Compatibility with the Elastic Common Schema (ECS)
editThis plugin can be used to decode CEF events into the Elastic Common Schema, or to encode ECS-compatible events into CEF. It can also be used without ECS, encoding and decoding events using only CEF-defined field names and keys.
The ECS Compatibility mode for a specific plugin instance can be controlled by setting ecs_compatibility when defining the codec:
input {
tcp {
# ...
codec => cef {
ecs_compatibility => v1
}
}
}
If left unspecified, the value of the pipeline.ecs_compatibility setting is used.
Timestamps and ECS compatiblity
editWhen decoding in ECS Compatibility Mode, timestamp-type fields are parsed and normalized to specific points on the timeline.
Because the CEF format allows ambiguous timestamp formats, some reasonable assumptions are made:
- When the timestamp does not include a year, we assume it happened in the recent past (or very near future to accommodate out-of-sync clocks and timezone offsets).
-
When the timestamp does not include UTC-offset information, we use the event’s
timezone (
dtzordeviceTimeZonefield), or fall through to this plugin’sdefault_timezone. -
Localized timestamps are parsed using the provided
locale.
Field mapping
editThe header fields from each CEF payload is expanded to the following fields, depending on whether ECS is enabled.
Header field mapping
edit| ECS Disabled | ECS Field |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
When decoding CEF payloads with ecs_compatibility => disabled, the abbreviated CEF Keys found in extensions are expanded, and CEF Field Names are inserted at the root level of the event.
When decoding in an ECS Compatibility mode, the ECS Fields are populated from the corresponding CEF Field Names or CEF Keys found in the payload’s extensions.
The following is a mapping between these fields.
Extension field mapping
edit| CEF Field Name (optional CEF Key) | ECS Field |
|---|---|
|
|
|
Multiple possible CEF fields map to this ECS Field. When decoding, the last entry encountered wins. When encoding, this field has higher priority. |
|
|
|
|
|
|
|
Multiple possible CEF fields map to this ECS Field. When decoding, the last entry encountered wins. When encoding, this field has lower priority. |
|
This field contains a timestamp. In ECS Compatibility Mode, it is parsed to a specific point in time. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Multiple possible CEF fields map to this ECS Field. When decoding, the last entry encountered wins. When encoding, this field has higher priority. |
|
|
|
|
|
|
|
|
|
Multiple possible CEF fields map to this ECS Field. When decoding, the last entry encountered wins. When encoding, this field has lower priority. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
When plugin configured with |
When plugin configured with |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
When plugin configured with |
When plugin configured with |
|
|
|
|
When plugin configured with |
When plugin configured with |
|
|
|
|
When plugin configured with |
When plugin configured with |
|
|
|
|
When plugin configured with |
When plugin configured with |
|
|
|
|
|
|
|
|
|
|
|
|
This field contains a timestamp. In ECS Compatibility Mode, it is parsed to a specific point in time. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This field contains a timestamp. In ECS Compatibility Mode, it is parsed to a specific point in time. |
|
|
|
|
|
|
|
|
|
|
|
|
|
This field contains a timestamp. In ECS Compatibility Mode, it is parsed to a specific point in time. |
|
|
|
|
|
|
|
|
|
|
|
This field contains a timestamp. In ECS Compatibility Mode, it is parsed to a specific point in time. |
|
|
|
This field contains a timestamp. In ECS Compatibility Mode, it is parsed to a specific point in time. |
|
|
|
|
|
This field contains a timestamp. In ECS Compatibility Mode, it is parsed to a specific point in time. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Multiple possible CEF fields map to this ECS Field. When decoding, the last entry encountered wins. When encoding, this field has higher priority. |
|
|
|
|
|
|
|
|
|
Multiple possible CEF fields map to this ECS Field. When decoding, the last entry encountered wins. When encoding, this field has lower priority. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This field contains a timestamp. In ECS Compatibility Mode, it is parsed to a specific point in time. |
|
|
|
|
Cef Codec Configuration Options
edit| Setting | Input type | Required |
|---|---|---|
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
||
No |
default_timezone
edit- Value type is string
-
Supported values are:
-
Timezone names (such as
Europe/Moscow,America/Argentina/Buenos_Aires) -
UTC Offsets (such as
-08:00,+03:00)
-
Timezone names (such as
- The default value is your system time zone
- This option has no effect when encoding.
When parsing timestamp fields in ECS mode and encountering timestamps that
do not contain UTC-offset information, the deviceTimeZone (dtz) field
from the CEF payload is used to interpret the given time. If the event does
not include timezone information, this default_timezone is used instead.
delimiter
edit- Value type is string
- There is no default value for this setting.
If your input puts a delimiter between each CEF event, you’ll want to set this to be that delimiter.
For example, with the TCP input, you probably want to put this:
input {
tcp {
codec => cef { delimiter => "\r\n" }
# ...
}
}
This setting allows the following character sequences to have special meaning:
-
\r(backslash "r") - means carriage return (ASCII 0x0D) -
\n(backslash "n") - means newline (ASCII 0x0A)
device
edit- Value type is string
-
Supported values are:
-
observer: indicates that device-specific fields represent the device used to observe the event. -
host: indicates that device-specific fields represent the device on which the event occurred.
-
-
The default value for this setting is
observer. -
Option has no effect when
ecs_compatibility => disabled. - Option has no effect when encoding
Defines a set of device-specific CEF fields as either representing the device on which an
event occurred, or merely the device from which the event was observed.
This causes the relevant fields to be routed to either the host or the observer
top-level groupings.
If the codec handles data from a variety of sources, the ECS recommendation is to use observer.
ecs_compatibility
edit- Value type is string
-
Supported values are:
-
disabled: uses CEF-defined field names in the event (e.g.,bytesIn,sourceAddress) -
v1: supports ECS-compatible event fields (e.g.,[source][bytes],[source][ip])
-
-
Default value depends on which version of Logstash is running:
-
When Logstash provides a
pipeline.ecs_compatibilitysetting, its value is used as the default -
Otherwise, the default value is
disabled.
-
When Logstash provides a
Controls this plugin’s compatibility with the Elastic Common Schema (ECS).
fields
edit- Value type is array
-
Default value is
[] - Option has no effect when decoding
When this codec is used in an Output Plugin, a list of fields can be provided to be included in CEF extensions part as key/value pairs.
locale
edit- Value type is string
-
Supported values are:
-
Abbreviated language_COUNTRY format (e.g.,
en_GB,pt_BR) -
Valid IETF BCP 47 language tag (e.g.,
zh-cmn-Hans-CN)
-
Abbreviated language_COUNTRY format (e.g.,
- The default value is your system locale
- Option has no effect when encoding
When parsing timestamp fields in ECS mode and encountering timestamps in
a localized format, this locale is used to interpret locale-specific strings
such as month abbreviations.
name
edit- Value type is string
-
Default value is
"Logstash" - Option has no effect when decoding
When this codec is used in an Output Plugin, this option can be used to specify the
value of the name field in the CEF header. The new value can include %{foo} strings
to help you build a new value from other parts of the event.
product
edit- Value type is string
-
Default value is
"Logstash" - Option has no effect when decoding
When this codec is used in an Output Plugin, this option can be used to specify the
value of the device product field in CEF header. The new value can include %{foo} strings
to help you build a new value from other parts of the event.
reverse_mapping
edit- Value type is boolean
-
Default value is
false - Option has no effect when decoding
Set to true to adhere to the specifications and encode using the CEF key name (short name) for the CEF field names.
severity
edit- Value type is string
-
Default value is
"6" - Option has no effect when decoding
When this codec is used in an Output Plugin, this option can be used to specify the
value of the severity field in CEF header. The new value can include %{foo} strings
to help you build a new value from other parts of the event.
Defined as field of type string to allow sprintf. The value will be validated to be an integer in the range from 0 to 10 (including). All invalid values will be mapped to the default of 6.
signature
edit- Value type is string
-
Default value is
"Logstash" - Option has no effect when decoding
When this codec is used in an Output Plugin, this option can be used to specify the
value of the signature ID field in CEF header. The new value can include %{foo} strings
to help you build a new value from other parts of the event.
vendor
edit- Value type is string
-
Default value is
"Elasticsearch" - Option has no effect when decoding
When this codec is used in an Output Plugin, this option can be used to specify the
value of the device vendor field in CEF header. The new value can include %{foo} strings
to help you build a new value from other parts of the event.
version
edit- Value type is string
-
Default value is
"1.0" - Option has no effect when decoding
When this codec is used in an Output Plugin, this option can be used to specify the
value of the device version field in CEF header. The new value can include %{foo} strings
to help you build a new value from other parts of the event.