Beats version 6.3.0

edit

View commits

Breaking changes

edit

Affecting all Beats

  • De dot keys of labels and annotations in kubernetes meta processors to prevent collisions. 6203
  • Rename beat.cpu.*.time metrics to beat.cpu.*.time.ms. 6449
  • Add host.name field to all events, to avoid mapping conflicts. This could be breaking Logstash configs if you rely on the host field being a string. 7051

Filebeat

  • Add validation for Stdin, when Filebeat is configured with Stdin and any other inputs, Filebeat will now refuse to start. 6463
  • Mark system.syslog.message and system.auth.message as text instead of keyword. 6589

Metricbeat

  • De dot keys in kubernetes/event metricset to prevent collisions. 6203
  • Add config option for windows/perfmon metricset to ignore non existent counters. 6432
  • Refactor docker CPU calculations to be more consistent with docker stats. 6608
  • Update logstash.node_stats metricset to write data under logstash.node.stats.*. 6714

Bugfixes

edit

Affecting all Beats

  • Fix panic when Events containing a float32 value are normalized. 6129
  • Fix setup.dashboards.always_kibana when using Kibana 5.6. 6090
  • Fix for Kafka logger. 6430
  • Remove double slashes in Windows service script. 6491
  • Ensure Kubernetes labels/annotations don’t break mapping 6490
  • Ensure that the dashboard zip files can’t contain files outside of the kibana directory. 6921
  • Fix map overwrite panics by cloning shared structs before doing the update. 6947
  • Fix delays on autodiscovery events handling caused by blocking runner stops. 7170
  • Do not emit Kubernetes autodiscover events for Pods without IP address. 7235
  • Fix self metrics when containerized 6641

Auditbeat

  • Add hex decoding for the name field in audit path records. 6687
  • Fixed a deadlock in the file_integrity module under Windows. 6864
  • Fixed parsing of AppArmor audit messages. 6978
  • Allow auditbeat setup to run without requiring elevated privileges for the audit client. 7111
  • Fix goroutine leak that occurred when the auditd module was stopped. 7163

Filebeat

  • Fix panic when log prospector configuration fails to load. 6800
  • Fix memory leak in log prospector when files cannot be read. 6797
  • Add raw JSON to message field when JSON parsing fails. 6516
  • Commit registry writes to stable storage to avoid corrupt registry files. 6877
  • Fix a parsing issue in the syslog input for RFC3339 timestamp and time with nanoseconds. 7046
  • Fix an issue with an overflowing wait group when using the TCP input. 7202

Heartbeat

  • Fix race due to updates of shared a map, that was not supposed to be shared between multiple go-routines. 6616

Metricbeat

  • Fix the default configuration for Logstash to include the default port. 6279
  • Fix dealing with new process status codes in Linux kernel 4.14+. 6306
  • Add filtering option by exact device names in system.diskio. diskio.include_devices. 6085
  • Add connections metricset to RabbitMQ module 6548
  • Fix panic in http dependent modules when invalid config was used. 6205
  • Fix system.filesystem.used.pct value to match what df reports. 5494
  • Fix namespace disambiguation in Kubernetes state_* metricsets. 6281
  • Fix Windows perfmon metricset so that it sends metrics when an error occurs. 6542
  • Fix Kubernetes calculated fields store. https://github.com/elastic/beats/pull/6564{6564}
  • Exclude bind mounts in fsstat and filesystem metricsets. 6819
  • Don’t stop Metricbeat if aerospike server is down. 6874
  • disk reads and write count metrics in RabbitMQ queue metricset made optional. 6876
  • Add mapping for docker metrics per cpu. 6843

Winlogbeat

  • Fixed a crash under Windows 2003 and XP when an event had less insert strings than required by its format string. 6247
  • Fix config validation to allow event_logs.processors. [pull]6217[6217]

Added

edit

Affecting all Beats

  • Update Golang 1.9.4 6326
  • Add the ability to log to the Windows Event Log. 5913
  • The node name can be discovered automatically by machine-id matching when beat deployed outside Kubernetes cluster. 6146
  • Panics will be written to the logger before exiting. 6199
  • Add builder support for autodiscover and annotations builder 6408
  • Add plugin support for autodiscover builders, providers 6457
  • Preserve runtime from container statuses in Kubernetes autodiscover 6456
  • Experimental feature setup.template.append_fields added. 6024
  • Add appender support to autodiscover 6469
  • Add add_host_metadata processor 5968
  • Retry configuration to load dashboards if Kibana is not reachable when the beat starts. 6560
  • Add has_fields conditional to filter events based on the existence of all the given fields. 6285 6653
  • Add support for spooling to disk to the beats event publishing pipeline. 6581
  • Added logging of system info at Beat startup. 5946
  • Do not log errors if X-Pack Monitoring is enabled but Elastisearch X-Pack is not. 6627
  • Add rename processor. 6292
  • Allow override of dynamic template match_mapping_type for fields with object_type. 6691

Filebeat

  • Add IIS module to parse access log and error log. 6127
  • Renaming of the prospector type to the input type and all prospectors are now moved to the input folder, to maintain backward compatibility type aliasing was used to map the old type to the new one. This change also affect YAML configuration. 6078
  • Addition of the TCP input 6700
  • Add option to convert the timestamps to UTC in the system module. 5647
  • Add Logstash module support for main log and the slow log, support the plain text or structured JSON format 5481
  • Add stream filtering when using docker prospector. 6057
  • Add support for CRI logs format. 5630
  • Add json.ignore_decoding_error config to not log json decoding erors. 6547
  • Make registry file permission configurable. 6455
  • Add MongoDB module. 6238
  • Add Ingest pipeline loading to setup. 6814
  • Add support of log_format combined to NGINX access logs. 6858
  • Release config reloading feature as GA.
  • Add support human friendly size for the UDP input. 6886
  • Add Syslog input to ingest RFC3164 Events via TCP and UDP 6842
  • Remove the undefined username option from the Redis input and clarify the documentation. 6662

Heartbeat

  • Made the URL field of Heartbeat aggregateable. 6263
  • Use match.Matcher for checking Heartbeat response bodies with regular expressions. 6539

Metricbeat

  • Support apache status pages for versions older than 2.4.16. 6450
  • Add support for huge pages on Linux. 6436
  • Support to optionally de dot keys in http/json metricset to prevent collisions. 5970
  • Add graphite protocol metricbeat module. 4734
  • Add http server metricset to support push metrics via http. 4770
  • Make config object public for graphite and http server 4820
  • Add system uptime metricset. 4848
  • Add experimental queue metricset to RabbitMQ module. 4788
  • Add additional php-fpm pool status kpis for Metricbeat module 5287
  • Add etcd module. 4970
  • Add ip address of docker containers to event. 5379
  • Add ceph osd tree information to metricbeat 5498
  • Add ceph osd_df to metricbeat 5606
  • Add basic Logstash module. 5540
  • Add dashboard for Windows service metricset. 5603
  • Add pct calculated fields for Pod and container CPU and memory usages. 6158
  • Add statefulset support to Kubernetes module. 6236
  • Refactor prometheus endpoint parsing to look similar to upstream prometheus 6332
  • Making the http/json metricset GA. 6471
  • Add support for array in http/json metricset. 6480
  • Making the jolokia/jmx module GA. 6143
  • Making the MongoDB module GA. 6554
  • Allow to disable labels dedot in Docker module, in favor of a safe way to keep dots. 6490
  • Add experimental module to collect metrics from munin nodes. 6517
  • Add support for wildcards and explicit metrics grouping in jolokia/jmx. 6462
  • Set collector as default metricset in Prometheus module. 6636 6747
  • Set mntr as default metricset in Zookeeper module. 6674
  • Set default metricsets in vSphere module. 6676
  • Set status as default metricset in Apache module. 6673
  • Set namespace as default metricset in Aerospike module. 6669
  • Set service as default metricset in Windows module. 6675
  • Set all metricsets as default metricsets in uwsgi module. 6688
  • Allow autodiscover to monitor unexposed ports 6727
  • Mark kubernetes.event metricset as beta. 6715
  • Set all metricsets as default metricsets in couchbase module. 6683
  • Mark uwsgi module and metricset as beta. 6717
  • Mark Golang module and metricsets as beta. 6711
  • Mark system.raid metricset as beta. 6710
  • Mark http.server metricset as beta. 6712
  • Mark metricbeat logstash module and metricsets as beta. 6713
  • Set all metricsets as default metricsets in Ceph module. 6676
  • Set container, cpu, diskio, healthcheck, info, memory and network in docker module as default. 6718
  • Set cpu, load, memory, network, process and process_summary as default metricsets in system module. 6689
  • Set collector as default metricset in Dropwizard module. 6669
  • Set info and keyspace as default metricsets in redis module. 6742
  • Set connection as default metricset in rabbitmq module. 6743
  • Set all metricsets as default metricsets in Elasticsearch module. 6755
  • Set all metricsets as default metricsets in Etcd module. 6756
  • Set server metricsets as default in Graphite module. 6757
  • Set all metricsets as default metricsets in HAProxy module. 6758
  • Set all metricsets as default metricsets in Kafka module. 6759
  • Set all metricsets as default metricsets in postgresql module. 6761
  • Set status metricsets as default in Kibana module. 6762
  • Set all metricsets as default metricsets in Logstash module. 6763
  • Set container, node, pod, system, volume as default in Kubernetes module. https://github.com/elastic/beats/pull/ 6764[6764]
  • Set stats as default in memcached module. 6765
  • Set all metricsets as default metricsets in Mongodb module. 6766
  • Set pool as default metricset for php_fpm module. 6768
  • Set status as default metricset for mysql module. https://github.com/elastic/beats/pull/ 6769[6769]
  • Set stubstatus as default metricset for nginx module. 6770
  • Added support for haproxy 1.7 and 1.8. 6793
  • Add accumulated I/O stats to diskio in the line of docker stats. 6701
  • Ignore virtual filesystem types by default in system module. 6819
  • Release config reloading feature as GA. 6891
  • Kubernetes deployment: Add ServiceAccount config to system metricbeat. 6824
  • Kubernetes deployment: Add DNS Policy to system metricbeat. 6656

Packetbeat

  • Add support for condition on bool type 5659 5954
  • Fix high memory usage on HTTP body if body is not published. 6680
  • Allow to capture the HTTP request or response bodies independently. 6784
  • HTTP publishes an Error event for unmatched requests or responses. 6794

Winlogbeat

  • Use bookmarks to persist the last published event. 6150