« Beats version 7.0.0-rc1 Beats version 7.0.0-alpha2 »
Elastic Docs ›Beats Platform Reference [8.16] ›Release notes

Beats version 7.0.0-beta1

edit

Beats version 7.0.0-beta1

edit

Check the HEAD diff

Breaking changes

edit

Affecting all Beats

  • Embedded html is not escaped anymore by default. 9914
  • Remove port settings from Logstash and Redis output. 9934
  • Rename process.exe to process.executable in add_process_metadata to align with ECS. 9949
  • Import ECS change ecs#308: leaf field user.group is now the group field set. 10275
  • Update the code of Central Management to align with the new returned format. 10019
  • Docker and Kubernetes labels/annotations will be "dedoted" by default. 10338
  • Remove --setup command line flag. 10138
  • Remove --version command line flag. 10138
  • Remove --configtest command line flag. 10138
  • Move output.elasticsearch.ilm settings to setup.ilm. 10347
  • ILM will be available by default if Elasticsearch > 7.0 is used. 10347

Auditbeat

  • Rename process.exe to process.executable in auditd module to align with ECS. 9949
  • Rename process.cwd to process.working_directory in auditd module to align with ECS. 10195
  • Change data type of process.pid and process.ppid to number in JSON output of the auditd module. 10195
  • Change data type of file.uid and file.gid to string in JSON output of the FIM module. 10195
  • Field file.origin changed type from text to keyword. 10544
  • Rename user fields to ECS in auditd module. 10456
  • Rename event.type to auditd.message_type in auditd module because event.type is reserved for future use by ECS. 10536
  • Rename auditd.messages to event.original and auditd.warnings to error.message. 10577

Filebeat

  • Rename many kibana.log.* fields to map to ECS. 9301
  • Modify apache/error dataset to follow ECS. 8963
  • Rename many traefik.access.* fields to map to ECS. 9005
  • Fix parsing of GC entries in elasticsearch server log. 9513 9810
  • Rename read_timestamp to event.created for Redis input. 9924
  • Rename a few elasticsearch.audit.* fields to map to ECS. 9293
  • Rename read_timestamp to event.created for all Filebeat modules using it. 10139
  • Rename many iis.error.* fields to map to ECS. 9955
  • Adjust fileset haproxy.log to map to ECS. 10143
  • Rename a few logstash.* fields to map to ECS, remove logstash.slowlog.message. 9935
  • Rename a few mongodb.* fields to map to ECS. 10009
  • Rename a few mysql.* fields to map to ECS. 10008
  • Rename a few nginx.error.* fields to map to ECS. 10007
  • Rename many auditd.log.* fields to map to ECS. 10192
  • Filesets with multiple ingest pipelines added in 8914 only work with Elasticsearch >= 6.5.0 10001
  • Remove service.name from Elastcsearch module. Replace by service.type. 10042
  • Remove numeric coercions for user.id and group.id. IDs should be keyword. 10233
  • Add grok pattern to support redis 5.0.3 log timestamp. 9819 10033
  • Now save the first seen timestamp in event.created (previously read_timestamp), instead of saving the parsed date. Now aligned with event.created semantics elsewhere. 10139
  • Rename mysql.error.thread_id and mysql.slowlog.id to mysql.thread_id. 10161
  • Remove mysql.error.timestamp and mysql.slowlog.timestamp. 10161
  • Migrate multiple fields to event.duration, from modules "apache", "elasticsearch", "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik", including http.response.elapsed_time (ECS). 10188, 10274
  • Rename multiple fields to http.response.body.bytes, from modules "apache", "iis", "kibana", "nginx" and "traefik", including http.response.content_length (ECS). 10188
  • Change type from haproxy.log fileset fields from text to keyword: response.captured_headers, request.captured_headers, raw_request_line, mode. 10397
  • Change type of field backend_url and frontend_name in traefik.access metricset to type keyword. 10401
  • Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above 10352
  • Migrate Elasticsearch audit logs fields to ECS 10352
  • Several text fields in the Logstash module are now indexed as keyword fields with text multi-fields (ECS). 10417
  • Several text fields in the Elasticsearch module are now indexed as keyword fields with text multi-fields (ECS). 10414
  • Move dissect pattern for traefik.access fileset from Filbeat to Elasticsearch. 10442
  • The elasticsearch/deprecation fileset now indexes the component field under elasticsearch instead of elasticsearch.server. 10445
  • Remove field kafka.log.trace.full from kafka.log fielset. 10398
  • Change field kafka.log.class for kafka.log fileset from text to keyword. 10398
  • Address add_kubernetes_metadata processor issue where old source field is still used for matcher. 10505 10506
  • Change type of haproxy.source from text to keyword. 10506
  • Rename event.type to suricata.eve.event_type in Suricata module because event.type is reserved for future use by ECS. 10575
  • Populate more ECS fields in the Suricata module. 10006
  • Rename setting filebeat.registry_flush to filebeat.registry.flush. 10504
  • Rename setting filebeat.registry_file_permission to filebeat.registry.file_permission. 10504
  • Remove setting filebeat.registry_file in favor of filebeat.registry.path. The registry file will be stored in a sub-directory by now. 10504

Heartbeat

  • Remove monitor generator script that was rarely used. 9648
  • monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values. If you wish to have continuity with the old format of monitor IDs you’ll need to set the id property explicitly. 9697
  • A number of fields have been aliased to their relevant counterparts in the url.* field. Existing visualizations should mostly work. The fields that have been moved are monitor.scheme -> url.scheme, monitor.host -> url.domain, resolve.host -> url.domain, http.url -> url.full, tcp.port -> url.port. In addition to these moves the new fields url.username, url.password, url.path, and url.query are now present. It should be noted that the url.password field does not contain actual password values, but rather the text <hidden> 9570.
  • The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. 10294

Journalbeat

  • Rename read_timestamp to event.created to align with ECS. 10043, 10139
  • Rename host.name to host.hostname to align with ECS. 10043
  • Fix typo in the field name container.id_truncated. 10525
  • Rename container.image.tag to container.log.tag. 10561
  • Change type of text fields to keyword. 10542

Metricbeat

  • Migrate system process metricset fields to ECS. 10332
  • Refactor Prometheus metric mappings 9948
  • Removed Prometheus stats metricset in favor of just using Prometheus collector 9948
  • Migrate system socket metricset fields to ECS. 10339
  • Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. 10339
  • Adjust Redis.info metricset fields to ECS. 10319
  • Change type of field docker.container.ip_addresses to ip instead of keyword. 10364
  • Rename http.request.body field to http.request.body.content. 10315
  • Adjust php_fpm.process metricset fields to ECS. 10366
  • Adjust mongodb.status metricset to to ECS. 10368
  • Refactor munin module to collect an event per plugin and to have more strict field mappings. namespace option has been removed, and will be replaced by service.name. 10322
  • Change the following fields from type text to keyword: 10318
  • ceph.osd_df.name
  • ceph.osd_tree.name
  • ceph.osd_tree.children
  • kafka.consumergroup.meta
  • kibana.stats.name
  • mongodb.metrics.replication.executor.network_interface
  • php_fpm.process.request_uri
  • php_fpm.process.script
  • Add service.name option to all modules to explicitly set service.name if it is unset. 10427
  • Update a few elasticsearch.* fields to map to ECS. 10350
  • Update a few logstash.* fields to map to ECS. 10350
  • Update a few kibana.* fields to map to ECS. 10350
  • Update rabbitmq.* fields to map to ECS. 10563
  • Update haproxy.* fields to map to ECS. 10558 10568
  • Collect all EC2 meta data from all instances in all states. 10628
  • Fix MongoDB dashboard that had some incorrect field names from status Metricset 9795 9715

Packetbeat

  • Adjust Packetbeat http fields to ECS Beta 2 9645
  • http.request.body moves to http.request.body.content
  • http.response.body moves to http.response.body.content
  • Changed Packetbeat fields to align with ECS. 7968
  • Removed trailing dot from domain names reported by the DNS protocol. 9941

Winlogbeat

  • Adjust Winlogbeat fields to map to ECS. 10333

Functionbeat

  • Correctly normalize Cloudformation resource name. 10087
  • Functionbeat can now deploy a function for Kinesis. {10116}10116[10116]
  • Allow functionbeat to use the keystore. 9009

Bugfixes

edit

Affecting all Beats

  • Fix config appender registration. 9873
  • Gracefully handle TLS options when enrolling a Beat. 9129
  • The backing off now implements jitter to better distribute the load. 10172
  • Fix TLS certificate DoS vulnerability. 10302
  • Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289
  • Fix encoding of timestamps when using disk spool. 10099
  • Fix stopping of modules started by kubernetes autodiscover. 10476
  • Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587
  • Fix unauthorized error when loading dashboards by adding username and password into kibana config. 10513 10675
  • Fix exclude_labels when there are dotted keys 10154
  • Fix registry handle leak on Windows (https://github.com/elastic/go-sysinfo/pull/33). 9920

Auditbeat

  • Enable System module config on Windows. 10237

Filebeat

  • Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869, access log: 9955.
  • Support haproxy log lines without captured headers. 9463 9958
  • Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135
  • Fix bad bytes count in docker input when filtering by stream. 10211
  • Fixed data types for roles and indices fields in elasticsearch/audit fileset 10307
  • Ensure source.address is always populated by the nginx module (ECS). 10418
  • Support mysql 5.7.22 slowlog starting with time information. 7892 9647

Heartbeat

  • Made monitors.d configuration part of the default config. 9004
  • Fixed rare issue where TLS connections to endpoints with x509 certificates missing either notBefore or notAfter would cause the check to fail with a stacktrace. 9566

Journalbeat

  • Do not stop collecting events when journal entries change. 9994

Metricbeat

  • Fix panics in vsphere module when certain values where not returned by the API. 9784
  • Fix pod UID metadata enrichment in Kubernetes module. 10081
  • Fix issue that would prevent collection of processes without command line on Windows. 10196
  • Fixed data type for tags field in docker/container metricset 10307
  • Fixed data type for tags field in docker/image metricset 10307
  • Fixed data type for isr field in kafka/partition metricset 10307
  • Fixed data types for various hosts fields in mongodb/replstatus metricset 10307
  • Added function to close sql database connection. 10355
  • Fix issue with elasticsearch/node_stats metricset (x-pack) not indexing source_node field. 10639

Packetbeat

  • Fix DHCPv4 dashboard that wouldn’t load in Kibana. 9850
  • Fixed a crash when using af_packet capture 10477

Winlogbeat

  • Close handle on signalEvent. 9838

Functionbeat

  • Ensure that functionbeat is logging at info level not debug. 10262
  • Add the required permissions to the role when deployment SQS functions. 9152

Added

edit

Affecting all Beats

  • Update field definitions for http to ECS Beta 2 9645
  • Add agent.id and agent.ephemeral_id fields to all beats. 9404
  • Add name config option to add_host_metadata processor. 9943
  • Add add_labels and add_tags processors. 9973
  • Add missing file encoding to readers. 10080
  • Introduce migration.enabled configuration. 9805
  • Add alias field support in Kibana index pattern. 10075
  • Add add_fields processor. 10119
  • Add Kibana field formatter to bytes fields. 10184
  • Document a few more auditd.log.* fields. 10192
  • Support Kafka 2.1.0. 10440
  • Add ILM mode auto to setup.ilm.enabled setting. This new default value detects if ILM is available 10347
  • Add support to read ILM policy from external JSON file. 10347
  • Add overwrite and check_exists settings to ILM support. 10347
  • Generate Kibana index pattern on demand instead of using a local file. 10478
  • Calls to Elasticsearch X-Pack APIs made by Beats won’t cause deprecation logs in Elasticsearch logs. {9656}9656[9656]
  • Allow to unenroll a Beat from the UI. 9452
  • Release Jolokia autodiscover as GA. 9706
  • Allow Central Management to send events back to kibana. 9382

Auditbeat

  • Add system module. 9546
  • Add user.id (UID) and user.name for ECS. 10195
  • Add group.id (GID) and group.name for ECS. 10195
  • System module process dataset: Add user information to processes. 9963
  • Add system package dataset. 10225
  • Add system module login dataset. 9327
  • Add entity_id fields. 10500
  • Add seven dashboards for the system module. 10511

Filebeat

  • Add convert_timezone option to Elasticsearch module to convert dates to UTC. 9756 9761
  • Added module for parsing Google Santa logs. 9540
  • Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399
  • Add option to modules.yml file to indicate that a module has been moved 9432.
  • Add support for ssl_request_log in apache2 module. 8088 9833
  • Add support for iis 7.5 log format. 9753 9967
  • Add service.type field to all Modules. By default the field is set with the module name. It can be overwritten with service.type config. 10042
  • Add support for MariaDB in the slowlog fileset of mysql module. 9731
  • Apache module’s error fileset now performs GeoIP lookup, like the access fileset. 10273
  • Elasticsearch module’s slowlog now populates event.duration (ECS). 9293
  • HAProxy module now populates event.duration and http.response.bytes (ECS). 10143
  • Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137
  • Add convert_timezone to nginx module. 9839 10148
  • Add support for Percona in the slowlog fileset of mysql module. 6665 10227
  • Added support for ingesting structured Elasticsearch audit logs 10352
  • Added support for ingesting structured Elasticsearch slow logs 10445
  • Added support for ingesting structured Elasticsearch deprecation logs 10445
  • New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176
  • Added support for ingesting structured Elasticsearch server logs 10428
  • Populate more ECS fields in the Suricata module. 10006
  • Add module zeek. 9931 10034

Heartbeat

  • Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the docker key. 10258

Journalbeat

  • Migrate registry from previously incorrect path. 10486

Metricbeat

  • Add key metricset to the Redis module. 9582 9657 9746
  • Add socket_summary metricset to system defaults, removing experimental tag and supporting Windows 9709
  • Add docker event metricset. 9856
  • Add performance metricset to x-pack mssql module 9826
  • Add DeDot for kubernetes labels and annotations. 9860 9939
  • Add more meaningful metrics to performance Metricset on MSSQL module 10011
  • Rename some fields in performance Metricset on MSSQL module to match the updated documentation from Microsoft 10074
  • Add AWS EC2 module. 9257 9300
  • Release windows Metricbeat module as GA. 10163
  • Release traefik Metricbeat module as GA. 10166
  • Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094
  • List filesystems on Windows that have an access path but not an assigned letter 8916 10196
  • Add nats module. 10071
  • Release uswgi Metricbeat module GA. 10164
  • Release php_fpm module as GA. 10198
  • Release Memcached module as GA. 10199
  • Release etcd module as GA. 10200
  • Release Ceph module as GA. 10202
  • Release aerospike module as GA. 10203
  • Release kubernetes apiserver and event metricsets as GA 10212
  • Release Couchbase module as GA. 10201
  • Release RabbitMQ module GA. 10165
  • Release envoyproxy module GA. 10223
  • Release mongodb.metrics and mongodb.replstatus as GA. 10242
  • Release mysql.galera_status as GA. 10242
  • Release postgresql.statement as GA. 10242
  • Release RabbitMQ Metricbeat module GA. 10165
  • Release Dropwizard module as GA. 10240
  • Release Graphite module as GA. 10240
  • Release kvm module as beta. 10279
  • Release http.server metricset as GA. 10240
  • Release Nats module as GA. 10281
  • Release munin module as GA. 10311
  • Release Golang module as GA. 10312
  • Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222
  • Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261
  • Rename db Metricset to transaction_log in MSSQL Metricbeat module 10109
  • Add process arguments and the path to its executable file in the system process metricset 10332
  • Added server Metricset to Zookeeper Metricbeat module 8938 10341
  • Release AWS module as GA. 10345
  • Add overview dashboard to Zookeeper Metricbeat module 10379

Packetbeat

  • Add network.community_id to Packetbeat flow events. 10061
  • Add aliases for flow fields that were renamed. 7968 10063
  • Add support to decode mysql prepare statement command. 8084

Functionbeat

  • Mark Functionbeat as GA. 10564
« Beats version 7.0.0-rc1 Beats version 7.0.0-alpha2 »