The following list are changes in 8.0.0 as compared to 7.17.0, and combines release notes from the 8.0.0-alpha1, -alpha2, -beta1, -rc1 and -rc2 releases.
-
Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema (ECS).
This mode is controlled individually with each plugin’s ecs_compatibility option, which defaults to the value of the Logstash pipeline.ecs_compatibility setting.
In Logstash 8, this compatibility mode will be on-by-default for all pipelines.
If you wish to lock in a pipeline’s behavior from Logstash 7.x before upgrading to Logstash 8,
you can set
pipeline.ecs_compatibility: disabled
to its definition inpipelines.yml
(or globally inlogstash.yml
). - Starting from Logstash 8.0, the minimum required version of Java to run Logstash is Java 11. By default, Logstash will run with the bundled JDK, which has been verified to work with each specific version of Logstash, and generally provides the best performance and reliability.
-
Support for using
JAVA_HOME
to override the path to the JDK that Logstash runs with has been removed for this release. In the8.x
release, users should set the value ofLS_JAVA_HOME
to the path of their preferred JDK if they wish to use a version other than the bundled JDK. The value ofJAVA_HOME
will be ignored. - The Java Execution Engine has been the default engine since Logstash 7.0, and works with plugins written in either Ruby or Java. Removal of the Ruby Execution Engine will not affect the ability to run existing pipelines. #12517
- We have added support for UTF-16 and other multi-byte-character when reading log files. #9702
-
Setting
config.field_reference.parser
has been removed. The Field Reference parser interprets references to fields in your pipelines and plugins. Its behavior was configurable in 6.x, and since 7.0 allowed only a single option:strict
. 8.0 no longer recognizes the setting, but maintains the same behavior as thestrict
setting. Logstash rejects ambiguous and illegal inputs as standard behavior. #12466
For a more detailed view of these changes please check Breaking changes in 8.0.
- As processing times speed up, millisecond granularity is not always enough. Inbound data increasingly has sub-millisecond granularity timestamps. The pull request #12797 allows the internal mechanisms of Logstash that hold moment-in-time data - such as the Logstash Event, the Persistent Queue, the Dead Letter Queue and JSON encoding/decoding - to have nanosecond granularity.
- We have added another flag to the Benchmark CLI to allow passing a data file with previously captured data to the custom test case. This feature allows users to run the Benchmark CLI in a custom test case with a custom config and a custom dataset. #12437
Logstash 8.0.0 includes the same versions of all bundled plugins as Logstash 7.17.0. If you upgrade to 7.17 before upgrading to 8.0 (as recommended), you won’t see any changes to plugin versions.
Clone Filter - 4.2.0
- Added support for ECS v8 as alias for ECS v1 #27
Geoip Filter - 7.2.11
-
Improved compatibility with the Elastic Common Schema #206
-
Added support for ECS’s composite
region_iso_code
(US-WA
), which replaces the non-ECSregion_code
(WA
) as a default field with City databases. To get the stand-aloneregion_code
in ECS mode, you must include it in thefields
directive - [DOC] Improve ECS-related documentation
-
Added support for ECS’s composite
- [DOC] Air-gapped environment requires both ASN and City databases #204
Http Filter - 1.2.1
Ruby Filter - 3.1.8
- [DOC] Added doc to describe the option `tag_with_exception_message`https://github.com/logstash-plugins/logstash-filter-ruby/pull/62[#62]
- Fix SyntaxError handling so other pipelines can shut down gracefully #64
Useragent Filter - 3.3.3
- Docs: mention added fields in 3.3 with a note #78
Exec Input - 3.4.0
- Feat: adjust fields for ECS compatibility #28
-
Plugin will no longer override fields if they exist in the decoded payload (It no longer sets the
host
field if decoded from the command’s output)
Gelf Input - 3.3.1
-
Fix: safely coerce the value of
_@timestamp
to avoid crashing the plugin #67
Generator Input - 3.1.0
- Feat: adjusted fields for ECS compatibility #22
- Fix: do not override the host field if it’s present in the generator line (after decoding)
- Fix: codec flushing when closing input
Imap Input - 3.2.0
- Feat: ECS compatibility #55
-
added (optional)
headers_target
configuration option -
added (optional)
attachments_target
configuration option -
Fix: plugin should not close
$stdin
, while being stopped
Jms Input - 3.2.1
- Fix: improve compatibility with MessageConsumer implementations #51, such as IBM MQ.
-
Test: Fix test failures due to ECS compatibility default changes in
8.x
of logstash #53 - Feat: event_factory support + targets to aid ECS #49
- Fix: when configured to add JMS headers to the event, headers whose value is not set no longer result in nil entries on the event
-
Fix: when adding the
jms_reply_to
header to an event, a string representation is set instead of an opaque object.
Pipe Input - 3.1.0
- Feat: adjust fields for ECS compatibility #19
S3 Input - 3.8.3
Snmp Input - 1.3.1
Snmptrap Input - 3.1.0
- Feat: ecs_compatiblity support + (optional) target #37
Syslog Input - 3.6.0
- Add support for ECS v8 as alias to v1 implementation #68
Twitter Input - 4.1.0
- Feat: optional target + ecs_compatibility #72
Unix Input - 3.1.1
Jdbc Integration - 5.2.2
- Feat: name scheduler threads + redirect error logging #102
- Refactor: isolate paginated normal statement algorithm in a separate handler #101
-
Added
jdbc_paging_mode
option to choose if useexplicit
pagination in statements and avoid the initial count query or useauto
to delegate to the underlying library #95 -
Several improvements to Java driver loading
-
Refactor: to explicit Java (driver) class name loading #96. The change is expected to provide a more robust fix for the driver loading issue #83.
NOTE: A fatal driver error will no longer keep reloading the pipeline and now leads to a system exit.
- Fix: regression due returning the Java driver class #98
-
Kafka Integration - 10.9.0
-
Refactor: leverage codec when using schema registry
Previously using
schema_registry_url
parsed the payload as JSON even ifcodec => 'plain'
was explicitly set, this is no longer the case. #106
Cloudwatch Output - 3.0.10
- Fix: an old undefined method error which would surface with load (as queue fills up)
- Deps: unpin rufus scheduler #20
Elasticsearch Output - 11.4.1
-
Feat: upgrade manticore (http-client) library #1063
- the underlying changes include latest HttpClient (4.5.13)
-
resolves an old issue with
ssl_certificate_verification => false
still doing some verification logic
-
Updates ECS templates #1062
- Updates v1 templates to 1.12.1 for use with Elasticsearch 7.x and 8.x
- Updates BETA preview of ECS v8 templates for Elasticsearch 7.x and 8.x
- Feat: add support for traces data stream type #1057
- Refactor: review manticore error handling/logging, logging originating cause in case of connection related error when debug level is enabled. Java causes on connection related exceptions will now be extra logged when plugin is logging at debug level #1029
-
ECS-related fixes #1046
- Data Streams requirement on ECS is properly enforced when running on Logstash 8, and warned about when running on Logstash 7.
- ECS Compatibility v8 can now be selected
Core Patterns - 4.3.2
-
Fix: typo in
BIN9_QUERYLOG
pattern (in ECS mode) #307