Beats version 6.5.0

Affecting all Beats

Fixed add_host_metadata not initializing correctly on Windows. 7715
Fixed missing file unlock in spool file on Windows, so file can be reopened and locked. 7859
Fix spool file opening/creation failing due to file locking on Windows. 7859
Fix size of maximum mmaped read area in spool file on Windows. 7859
Fix potential data loss on OS X in spool file by using fcntl with F_FULLFSYNC. 7859
Improve fsync on linux, by assuming the kernel resets error flags of failed writes. 7859
Remove unix-like permission checks on Windows, so files can be opened. 7849
Replace index patterns in TSVB visualizations. 7929
Deregister pipeline loader callback when inputsRunner is stopped. 7893[7893]
Add backoff support to x-pack monitoring outputs. 7966
Removed execute permissions systemd unit file. 7873
Fix a race condition with the add_host_metadata and the event serialization. 8223 8653
Enforce that data used by k8s or docker doesn’t use any reference. 8240
Switch to different UUID lib due to to non-random generated UUIDs. 8485
Fix race condition when publishing monitoring data. 8646
Fix bug in loading dashboards from zip file. 8051
Fix in-cluster kubernetes configuration on IPv6. 8754
The export config subcommand should not display real value for field reference. 8769
The setup command will not fail if no dashboard is available to import. 8977
Fix central management configurations reload when a configuration is removed in Kibana. 9010

Auditbeat

Fixed a crash in the file_integrity module under Linux. 7753
Fixed the RPM by designating the config file as configuration data in the RPM spec. 8075
Fixed a concurrent map write panic in the auditd module. 8158
Fixed a data race in the file_integrity module. 8009
Fixed a deadlock in the file_integrity module. 8027

Filebeat

Fix date format in Mongodb Ingest pipeline. 7974
Fixed a docker input error due to the offset update bug in partial log join.https://github.com/elastic/beats/pull/8177[8177]
Update CRI format to support partial/full tags. 8265
Fix some errors happening when stopping syslog input. 8347
Fix RFC3339 timezone and nanoseconds parsing with the syslog input. 8346
Mark the TCP and UDP input as GA. 8125
Support multiline logs in logstash/log fileset of Filebeat. 8562
Support different timestamp format in postgresql module. 9494 9650

Heartbeat

Fixed bug where HTTP responses with larger bodies would incorrectly report connection errors. 8660

Metricbeat

Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. 7789
Fixed the RPM by designating the modules.d config files as configuration data in the RPM spec. 8075
Fixed the location of the modules.d dir in Deb and RPM packages. 8104
Add docker diskio stats on Windows. 6815 8126
Fix incorrect type conversion of average response time in Haproxy dashboards 8404
Added io disk read and write times to system module 8473 8508
Avoid mapping issues in kubernetes module. 8487
Recover metrics for old apache versions removed by mistake on #6450. 7871
Fix dropwizard module parsing of metric names. 8365 8385
Fix issue that would prevent kafka module to find a proper broker when port is not set 8613
Fix range colors in multiple visualizations. 8633 8634
Fix incorrect header parsing on http metricbeat module 8564 8585
Fixed a panic when the kvm module cannot establish a connection to libvirtd. 7792.

Packetbeat

Fixed a seccomp related error where the fcntl64 syscall was not permitted on 32-bit Linux and the sniffer failed to start. 7839
Added missing cmdline and client_cmdline fields to index template. 8258

Affecting all Beats

Added time-based log rotation. 8349
Add backoff on error support to redis output. 7781
Allow for cloud-id to specify a custom port. This makes cloud-id work in ECE contexts. 7887
Add support to grow or shrink an existing spool file between restarts. 7859
Make kubernetes autodiscover ignore events with empty container IDs 7971
Implement CheckConfig in RunnerFactory to make autodiscover check configs 7961
Add DNS processor with support for performing reverse lookups on IP addresses. 7770
Support for Kafka 2.0.0 in kafka output 8399
Add setting setup.kibana.space.id to support Kibana Spaces 7942
Better tracking of number of open file descriptors. 7986
Report number of open file handles on Windows. 8329
Added the add_process_metadata processor to enrich events with process information. 6789
Add Beats Central Management 8559
Report configured queue type. 8091
Enable host and cloud metadata processors by default. 8596

Filebeat

Add tag "truncated" to "log.flags" if incoming line is longer than configured limit. 7991
Add haproxy module. 8014
Add tag "multiline" to "log.flags" if event consists of multiple lines. 7997
Release docker input as GA. 8328
Keep unparsed user agent information in user_agent.original. 7832
Added default and TCP parsing formats to HAproxy module 8311 8637
Add Suricata IDS/IDP/NSM module. 8153 8693
Support for Kafka 2.0.0 8853

Heartbeat

Heartbeat is marked as GA.
Add automatic config file reloading. 8023
Added autodiscovery support 8415
Added support for extra TLS/x509 metadata. 7944
Added stats and state metrics for number of monitors and endpoints started. 8621
Add last monitor status to dashboard table. Further break out monitors in dashboard table by monitor.ip. 9022

Journalbeat

Metricbeat

Add replstatus metricset to MongoDB module 7604
Add experimental socket summary metricset to system module 6782
Move common kafka fields (broker, topic and partition.id) to the module level to facilitate events correlation 7767
Add fields for memory fragmentation, memory allocator stats, copy on write, master-slave status, and active defragmentation to info metricset of Redis module. 7695
Increase ignore_above for system.process.cmdline to 2048. 8100
Add support to renamed fields planned for redis 5.0. 8167
Allow TCP helper to support delimiters and graphite module to accept multiple metrics in a single payload. 8278
Added died PID state to process_system metricset on system module 8275
Add metrics metricset to MongoDB module. 7611
Added ccr metricset to Elasticsearch module. 8335
Support for Kafka 2.0.0 8399
Added support for query params in configuration 8286 8292
Add container image for docker metricsets. 8214 8438
Precalculate composed id fields for kafka dashboards. 8504
Add support for full status page output for php-fpm module as a separate metricset called process. 8394
Add Kafka dashboard. 8457
Release Kafka module as GA. 8854

Packetbeat

Functionbeat

Heartbeat

watch.poll_file is now deprecated and superceded by automatic config file reloading.

Metricbeat

Redis info replication.master_offset has been deprecated in favor of replication.master.offset.https://github.com/elastic/beats/pull/7695[7695]
Redis info clients fields longest_output_list and biggest_input_buf have been renamed to max_output_buffer and max_input_buffer based on the names they will have in Redis 5.0, both fields will coexist during a time with the same value 8167.
Move common kafka fields (broker, topic and partition.id) to the module level 7767.