Beats version 7.7.0
editBeats version 7.7.0
editBreaking changes
editAffecting all Beats
- Environment variables can no longer reference other environment variables or objects. 15937
-
Change
aws_elb
autodiscovery provider field name fromelb_listener.*
toaws.elb.*
. 16219 16402 -
Remove support for using
add_docker_metadata
andadd_kubernetes_metadata
processors from thescript
processor. They can still be used as normal processors in the configuration. 16349 16514
Bugfixes
editAffecting all Beats
- Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data. 17223
-
Fix
add_cloud_metadata
processor to better support modifying sub-fields with other processors. 13808 - Fix panic in the Logstash output when trying to send events to closed connection. 15568
- Fix logging target settings being ignored when Beats are started via systemd or docker. 12024 15442
- Fix issue where default go logger is not discarded when either * or stdout is selected. 10251 15708
-
Remove superfluous use of
number_of_routing_shards
setting from the default template. 16038 - Automatically convert index names to lowercase. 16081
- Fix loading processor annotation hints, allowing the value to be a full configuration section. 16348
-
Add
ssl.ca_sha256
to the list of supported TLS options. This option allows you to check that a specific certificate is used as part of the verified chain. 15717 -
Fix
NewContainerMetadataEnricher
to use default config for kubernetes module. No longer requires the user to havelabels.dedot: true
in the configuration as it is now properly the default. 16857 -
Improve logging messages for the
add_kubernetes_metadata
processor. 16866 - Fail to start if httpprof is used and it cannot be initialized. 17028
- Fix concurrency issues in convert processor when used in the global context. 17032
-
Fix bug with
monitoring.cluster_uuid
setting not always being exposed via GET /state Beats API. 16732 17420 -
Fix building on FreeBSD by removing build flags from
add_cloudfoundry_metadata
processor. 17486
Filebeat
- Fix mapping error when zeek weird logs do not contain IP addresses. 15906
- Fix merging of fileset inputs to replace paths and append processors. 16450
-
Fix Elasticsearch
_id
field set by S3 and Google Pub/Sub inputs. 17026 - Fix various Cisco FTD parsing issues. 16863 16889
- Fix default index pattern in IBM MQ Filebeat dashboard. 17146
- Fix a mapping exception when ingesting Logstash plain logs (7.4+) with pipeline ids containing non alphanumeric chars. 17242 17243
- Fix MySQL slowlog module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17086 17156
-
Fix
elasticsearch.audit
data ingest pipeline to be more forgiving with date formats found in Elasticsearch audit logs. 17406 - Fix decoding errors caused by trailing spaces in CEF messages. 17253
- Fix activemq module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17428
Metricbeat
-
Change
lookup_fields
setting frommetricset.host
toservice.address
. 15883 -
Make
logstash-xpack
module once again have parity with internally-collected Logstash monitoring data. 16198 -
Improve metrics collection in the
system/service
metricset on older linux distributions. 16902 - Use max in k8s apiserver dashboard aggregations. 17018
-
Check if CCR feature is available on Elasticsearch cluster before attempting to call CCR APIs from
elasticsearch/ccr
metricset. 16511 17073 - Use max in k8s overview dashboard aggregations. 17015
- Fix Disk Used and Disk Usage visualizations in the Metricbeat System dashboards. 12435 17272
- Fix missing Accept header for Prometheus and OpenMetrics module. 16870 17291
- Combine cloudwatch aggregated metrics into single event. 17345
- Fix how we filter services by name in system/service. 17400
-
Fix problem where
cloudwatch
metricset was not collecting tags correctly. 17419 17424 - Check if cpuOptions field is nil in DescribeInstances output in ec2 metricset. 17418
-
Fix
aws.s3.bucket.name
terms_field in s3 overview dashboard. 17542 - Fix Unix socket path in memcached module. 17512
- Fix vsphere VM dashboard host aggregation visualizations. 17555
Added
editAffecting all Beats
-
Include network information by default when using the
add_host_metadata
oradd_observer_metadata
processor. 15347 16077 -
Add
aws_ec2
provider for autodiscovery. 12518 14823 - Add support for multiple passwords in redis output. 16058 16206
- Add support for Histogram type in fields.yml. 16570
- Windows .exe files now have embedded file version info. 15232
-
Remove experimental flag from
setup.template.append_fields
. 16576 -
Add
add_cloudfoundry_metadata
processor to annotate events with Cloud Foundry application data. 16621 -
Add
translate_sid
processor on Windows for converting Windows security identifier (SID) values to names. 7451 16013 - Add support for Kubernetes provider to recognize namespace level defaults. 16321
-
Add ability to enrich the
container.id
with the process id by using theadd_process_metadata
processor. 15947 - Update RPM packages contained in Beat Docker images. 17035
- Add Kerberos support to Kafka input and output. 16781
Auditbeat
- Add examples to the kubernetes manifests to show how to configure the auditd module and use processors to enrich events with metadata.
- In the kubernetes manifests, mount the data directory from the host, so data persist between executions in the same node. 17429
- Log to stderr when using kubernetes manifests. 174443
- Fix memory leak on when we miss socket close kprobe events. 17500
Filebeat
- Add ECS tls fields to the smtp, rdp, and ssl filesets in the zeek module, and the s3access and elb filesets in the aws module. 15757 15936
-
Add Nginx
ingress_controller
fileset. 16197 - Add ECS tls and categorization fields to apache module. 16032 16121
- Add MQTT input. 15602 16204
- Improve ECS categorization, container, and process field mappings in auditd module. 16153 16280
- Add ECS categorization fields to activemq module. 16151 16201
- Improve ECS field mappings in aws module. 16154 16307
- Improve ECS categorization field mappings in googlecloud module. 16030 16500
-
Add
cloudwatch
andec2
filesets to aws module. 13716 16579 - Improve ECS categorization field mappings in kibana module. 16168 16652
-
Add
cloudfoundry
input to send events from Cloud Foundry. 16586 - Improve ECS field mappings in haproxy module. 16162 16529
- Allow users to override pipeline ID in fileset input config. 9531 16561
- Improve ECS categorization field mappings in logstash module. 16169 16668
- Improve ECS categorization field mappings in iis module. 16165 16618
-
Improve the
decode_cef
processor by reducing the number of memory allocations. 16587 - Improve ECS categorization field mapping in kafka module. 16167 16645
- Improve ECS categorization field mapping in icinga module. 16164 16533
- Improve ECS categorization field mappings in ibmmq module. 16163 16532
- Add custom string mapping to CEF module to support Forcepoint NGFW. 14663 15910
- Add ECS fields to CEF module. 16157 16338
- Improve ECS categorization and host field mappings in elasticsearch module. 16160 16469
- Improve ECS categorization field mappings in suricata module. 16181 16843
- Release ActiveMQ module as GA. 17047 17049
- Improve ECS categorization field mappings in iptables module. 16166 16637
- Add pattern for Cisco ASA / FTD Message 734001. 16212 16612
-
Add
o365audit
input type for consuming events from Office 365 Management Activity API. 16196 16244 - Add custom string mapping to CEF module to support Check Point devices. 16041 16907
-
Add
o365
module for ingesting Office 365 management activity API events. 16196 16386 - Add Okta module. 16362
- Improve AWS cloudtrail field mappings. 16086 16110 17155
-
Make the
azure-eventhub
input GA. 15671 17313 -
Add
access_key_id
,secret_access_key
, andsession_token
to the aws module config. 17456
Heartbeat
- Allow a list of status codes for HTTP checks. 15587
Journalbeat
-
Improve parsing of
syslog.pid
in Journalbeat to strip the username when present. 16116
Metricbeat
- Add lambda metricset in aws module. 15260
- Add DynamoDB AWS light module. 15097
- Add IBM MQ light-weight module. 15301
- Add mixer metricset for Istio Metricbeat module. 15696
- Add mesh metricset for Istio Metricbeat module. 15535
- Add pilot metricset for Istio Metricbeat module. 15761
- Add galley metricset for Istio Metricbeat module. 15857
-
Add
key/value
mode for SQL module. 15770 15845 - Add support for Unix socket in Memcached module. 13685 15822
-
Make the
system/cpu
metricset collect normalized CPU metrics by default. 15618 15729 - Add kubernetes storage class support via kube-state-metrics. 16145
-
Add
up
metric to prometheus metrics collected from host. 15948 - Add citadel metricset for Istio Metricbeat module. 15990
- Add support for processors in light modules. 14740 15923
- Add ability to collect AuroraDB metrics in rds metricset. 14142 16004
- Reuse connections in SQL module. 16001
-
Improve the
logstash
module (whenxpack.enabled
is set totrue
) to use the overridecluster_uuid
returned by Logstash APIs. 15772 15795 - Add region parameter in googlecloud module. 15780 16203
-
Add
database_account
azure metricset. 15758 - Add support for Dropwizard metrics 4.1. 16332
- Add support for NATS 2.1. 16317
- Add azure container metricset in order to monitor containers. 15751 16421
-
Improve the
haproxy
module to support metrics exposed via HTTPS. 14579 16333 - Add filtering option for prometheus collector. 16420
-
Add metricsets based on Ceph Manager Daemon to the
ceph
module. 7723 16254 - Add Load Balancing metricset to GCP. 15559
-
Release
statsd
module as GA. 16447 14280 - Add collecting tags and tags_filter for rds metricset in aws module. 16605 16358
- Add OpenMetrics module. 16596
-
Add
redisenterprise
module. 16482 15269 -
Add
cloudfoundry
module to send events from Cloud Foundry. 16671 - Add system/users metricset as beta. 16569
- Align fields to ECS and add more tests for the azure module. 16024 16754
- Add additional cgroup fields to docker/diskio. 16638
- Add overview dashboard for googlecloud compute metricset. 16534 16819
- Add Prometheus remote write endpoint. 16609
- Release STAN module as GA. 16980
- Add query metricset for prometheus module. 17104
- Release ActiveMQ module as GA. 17047 17049
- Add support for CouchDB v2. 16352 16455
- Add dashboards for the azure container metricsets. 17194
-
Separate the
vpc
metricset into three smaller metricsets:vpn
,transitgateway
, andnatgateway
. 16892 - Use Elasticsearch histogram type to store Prometheus histograms. 17061
- Allow to rate Prometheus counters when scraping them. 17061
- Release the Oracle module as GA. 14279 16833
- Add Storage metricsets to GCP module. 15598
- Release the vsphere module as GA. 15798 17119
- Add PubSub metricset to Google Cloud Platform module. 15536
-
Add dashboard for
redisenterprise
module. 16752 - Add dashboard for VSphere host cluster and virtual machine. 14135
- Add test for documented fields check for metricsets without a http input. 17315 17334
- Release the azure module as GA. 17319
- In the kubernetes manifests, mount the data directory from the host, so data persist between executions in the same node. 17429
- Release the CockroachDB module as GA. 32527
Packetbeat
Winlogbeat