Logstash 7.8.0 Release Notes
editLogstash 7.8.0 Release Notes
editNew features and improvements
editExpanded JDK ecosystem and platform support
editWe can be more flexible and responsive in supporting new JDKs and deprecating old ones, thanks to recent improvements to our test scripts infrastructure. This work and other JDK14 fixes pave the way for Logstash to support both AdoptOpenJDK 11 and 14 in the near future.
Logstash has introduced support for running on CentOS/RHEL 8.x and Ubuntu 20.04. We’ve added new JDK support for Zulu 11, AdoptOpenJDK 11, and Oracle/OpenJDK/AdoptOpenJDK 14.
JVM version info is covered in Getting Started with Logstash. The complete list of supported operating systems and JVMs is available in the support matrix.
Elasticsearch API key support
editSupport for API keys was added to Elasticsearch in 6.7.0. With 7.8.0 Logstash introduces support for Elasticsearch API keys in the Elasticsearch output plugin #934.
Authentication in Elasticsearch can be done in different ways, from LDAP to SAML and others. User/password authentication makes sense for discrete users accessing Elasticsearch. For machine-to-machine communication, API key access is more common. Check out Grant access using API keys for more information about using API keys with Logstash and Elasticsearch.
Support for API keys in the Elasticsearch input and filter plugins, and the monitoring and management features will be added in upcoming releases.
Proxy support for monitoring and centralized management
editMany of our users deploy Logstash and the Elastic Stack in segmented networks where one component may not be able to directly reach out to another or to the Internet. Logstash plugins, such as the elasticsearch, http and SNS outputs, support the configuration of proxy servers. Version 7.8.0 brings proxy support to monitoring and central management #11799.
Configure the proxy’s URL in your logstash.yml
file using
"xpack.monitoring.elasticsearch.proxy" (for monitoring) or
"xpack.management.elasticsearch.proxy" (for central management).
Performance improvements and notable issues fixed
editAnnouncement: Azure and Netflow module deprecation
editAzure and Netflow modules in Logstash have been deprecated and replaced by the Azure modules in Filebeat and Metricbeat, and the Netflow module in Filebeat. The Filebeat and Metricbeat modules are compliant with the Elastic Common Schema (ECS).
Known issue
editPerformance regression. A potential performance regression may affect some users. This issue can cause a slowdown on pipeline compilation when multiple large pipelines are in use. We believe the issue was introduced in 7.7.0. This issue is currently being tracked and investigated in #12031
This issue seems to be affecting only big pipeline installations (that is, big pipeline definitions when multiple pipelines are defined). Symptoms include increased startup time and the appearance that Logstash is not responding to input events.
If you believe this issue is affecting you, we recommended that you downgrade to 7.6.2 while we continue to investigate and provide a resolution.
Plugins
editCef Codec - 6.1.1
- Improved encoding performance, especially when encoding many extension fields #81
- Fixed CEF short to long name translation for ahost/agentHostName field, according to documentation #75
- Fixed support for deep dot notation #73
-
Removed obsolete
sev
anddeprecated_v1_fields
fields - Fixed minor doc inconsistencies (added reverse_mapping to options table, moved it to alpha order in option descriptions, fixed typo) #60
- Added reverse_mapping option, which can be used to make encoder compliant to spec #51
- Fix handling of malformed inputs that have illegal unescaped-equals characters in extension field values (restores behaviour from ⇐ v5.0.3 in some edge-cases) #56
- Fix bug in parsing headers where certain legal escape sequences could cause non-escaped pipe characters to be ignored.
- Fix bug in parsing extension values where a legal unescaped space in a field’s value could be interpreted as a field separator #54
-
Add explicit handling for extension key names that use array-like syntax that isn’t legal with the strict-mode field-reference parser (e.g.,
fieldname[0]
becomes[fieldname][0]
). - Fix handling of higher-plane UTF-8 characters in message body
-
move
sev
anddeprecated_v1_fields
fields from deprecated to obsolete - added mapping for outcome = eventOutcome from CEF whitepaper (ref:p26/39)
- changed rt from receiptTime to deviceReceiptTime (ref:p27/39)
- changed tokenizer to include additional fields (ad.fieldname)
-
Add
delimiter
setting. This allows the decoder to be used with inputs like the TCP input where event delimiters are used. - Implements the dictionary translation for abbreviated CEF field names from chapter Chapter 2: ArcSight Extension Dictionary page 3 of 39 of the CEF specification.
-
add
_cefparsefailure
tag on failed decode - breaking: Updated plugin to use new Java Event APIs
-
Switch in-place sub! to sub when extracting
cef_version
. new Logstash Java Event does not support in-place String changes. - Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash
- New dependency requirements for logstash-core for the 5.0 release
-
Implements
encode
with escaping according to the CEF specification -
Config option
sev
is deprecated, useseverity
instead. - Plugins were updated to follow the new shutdown semantic. This allows Logstash to instruct input plugins to terminate gracefully, instead of using Thread.raise on the plugins' threads. #3895
- Dependency on logstash-core update to 2.0
Elasticsearch Filter - 3.7.1
- Fix: solves an issue where non-ascii unicode values in a template were not handled correctly #128
File Input - 4.1.18
- Fix: release watched files on completion (in read-mode) #271
-
Added configuration setting
check_archive_validity
settings to enable gzipped files verification. Fixes: #261 -
[DOC] Added clarification for settings available with
read
mode #235 -
[DOC] Rearranged text and fixed formatting for
mode
setting #266
Syslog Input - 3.4.2
Tcp Input - 6.0.5
- Fix potential startup crash that could occur when multiple instances of this plugin were started simultaneously #155
Kafka Integration - 10.2.0
- Changed: config defaults to be aligned with Kafka client defaults #30
- updated kafka client (and its dependencies) to version 2.4.1 #16
-
added the input
client_rack
parameter to enable support for follower fetching -
added the output
partitioner
parameter for tuning partitioning strategy - Refactor: normalized error logging a bit - make sure exception type is logged
- Fix: properly handle empty ssl_endpoint_identification_algorithm #8
-
Refactor : made
partition_assignment_strategy
option easier to configure by accepting simple values from an enumerated set instead of requiring lengthy class paths #25
Elasticsearch Output - 10.5.1
File Output - 4.3.0