This documentation contains work-in-progress information for future Elastic Stack and Cloud releases. Use the version selector to view supported release docs. It also contains some Elastic Cloud serverless information. Check out our serverless docs for more details.
Beats version 7.0.0-beta1
editBeats version 7.0.0-beta1
editBreaking changes
editAffecting all Beats
- Embedded html is not escaped anymore by default. 9914
- Remove port settings from Logstash and Redis output. 9934
-
Rename
process.exe
toprocess.executable
in add_process_metadata to align with ECS. 9949 -
Import ECS change ecs#308:
leaf field
user.group
is now thegroup
field set. 10275 - Update the code of Central Management to align with the new returned format. 10019
- Docker and Kubernetes labels/annotations will be "dedoted" by default. 10338
- Remove --setup command line flag. 10138
- Remove --version command line flag. 10138
- Remove --configtest command line flag. 10138
- Move output.elasticsearch.ilm settings to setup.ilm. 10347
- ILM will be available by default if Elasticsearch > 7.0 is used. 10347
Auditbeat
-
Rename
process.exe
toprocess.executable
in auditd module to align with ECS. 9949 -
Rename
process.cwd
toprocess.working_directory
in auditd module to align with ECS. 10195 -
Change data type of
process.pid
andprocess.ppid
to number in JSON output of the auditd module. 10195 -
Change data type of
file.uid
andfile.gid
to string in JSON output of the FIM module. 10195 -
Field
file.origin
changed type fromtext
tokeyword
. 10544 - Rename user fields to ECS in auditd module. 10456
-
Rename
event.type
toauditd.message_type
in auditd module because event.type is reserved for future use by ECS. 10536 -
Rename
auditd.messages
toevent.original
andauditd.warnings
toerror.message
. 10577
Filebeat
-
Rename many
kibana.log.*
fields to map to ECS. 9301 - Modify apache/error dataset to follow ECS. 8963
-
Rename many
traefik.access.*
fields to map to ECS. 9005 - Fix parsing of GC entries in elasticsearch server log. 9513 9810
-
Rename
read_timestamp
toevent.created
for Redis input. 9924 -
Rename a few
elasticsearch.audit.*
fields to map to ECS. 9293 -
Rename
read_timestamp
toevent.created
for all Filebeat modules using it. 10139 -
Rename many
iis.error.*
fields to map to ECS. 9955 -
Adjust fileset
haproxy.log
to map to ECS. 10143 -
Rename a few
logstash.*
fields to map to ECS, remove logstash.slowlog.message. 9935 -
Rename a few
mongodb.*
fields to map to ECS. 10009 -
Rename a few
mysql.*
fields to map to ECS. 10008 -
Rename a few
nginx.error.*
fields to map to ECS. 10007 -
Rename many
auditd.log.*
fields to map to ECS. 10192 - Filesets with multiple ingest pipelines added in 8914 only work with Elasticsearch >= 6.5.0 10001
- Remove service.name from Elastcsearch module. Replace by service.type. 10042
-
Remove numeric coercions for
user.id
andgroup.id
. IDs should bekeyword
. 10233 - Add grok pattern to support redis 5.0.3 log timestamp. 9819 10033
-
Now save the first seen timestamp in
event.created
(previouslyread_timestamp
), instead of saving the parsed date. Now aligned withevent.created
semantics elsewhere. 10139 -
Rename
mysql.error.thread_id
andmysql.slowlog.id
tomysql.thread_id
. 10161 -
Remove
mysql.error.timestamp
andmysql.slowlog.timestamp
. 10161 -
Migrate multiple fields to
event.duration
, from modules "apache", "elasticsearch", "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik", includinghttp.response.elapsed_time
(ECS). 10188, 10274 -
Rename multiple fields to
http.response.body.bytes
, from modules "apache", "iis", "kibana", "nginx" and "traefik", includinghttp.response.content_length
(ECS). 10188 -
Change type from haproxy.log fileset fields from text to keyword: response.captured_headers, request.captured_headers,
raw_request_line
,mode
. 10397 - Change type of field backend_url and frontend_name in traefik.access metricset to type keyword. 10401
- Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above 10352
- Migrate Elasticsearch audit logs fields to ECS 10352
-
Several text fields in the Logstash module are now indexed as
keyword
fields withtext
multi-fields (ECS). 10417 -
Several text fields in the Elasticsearch module are now indexed as
keyword
fields withtext
multi-fields (ECS). 10414 - Move dissect pattern for traefik.access fileset from Filbeat to Elasticsearch. 10442
-
The
elasticsearch/deprecation
fileset now indexes thecomponent
field underelasticsearch
instead ofelasticsearch.server
. 10445 -
Remove field
kafka.log.trace.full
from kafka.log fielset. 10398 -
Change field
kafka.log.class
for kafka.log fileset from text to keyword. 10398 - Address add_kubernetes_metadata processor issue where old source field is still used for matcher. 10505 10506
- Change type of haproxy.source from text to keyword. 10506
-
Rename
event.type
tosuricata.eve.event_type
in Suricata module because event.type is reserved for future use by ECS. 10575 - Populate more ECS fields in the Suricata module. 10006
-
Rename setting
filebeat.registry_flush
tofilebeat.registry.flush
. 10504 -
Rename setting
filebeat.registry_file_permission
tofilebeat.registry.file_permission
. 10504 -
Remove setting
filebeat.registry_file
in favor offilebeat.registry.path
. The registry file will be stored in a sub-directory by now. 10504
Heartbeat
- Remove monitor generator script that was rarely used. 9648
-
monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values. If you wish to have continuity with the old format of monitor IDs you’ll need to set the
id
property explicitly. 9697 -
A number of fields have been aliased to their relevant counterparts in the
url.*
field. Existing visualizations should mostly work. The fields that have been moved aremonitor.scheme -> url.scheme
,monitor.host -> url.domain
,resolve.host -> url.domain
,http.url -> url.full
,tcp.port -> url.port
. In addition to these moves the new fieldsurl.username
,url.password
,url.path
, andurl.query
are now present. It should be noted that theurl.password
field does not contain actual password values, but rather the text<hidden>
9570. - The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. 10294
Journalbeat
Metricbeat
- Migrate system process metricset fields to ECS. 10332
- Refactor Prometheus metric mappings 9948
- Removed Prometheus stats metricset in favor of just using Prometheus collector 9948
- Migrate system socket metricset fields to ECS. 10339
- Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. 10339
- Adjust Redis.info metricset fields to ECS. 10319
-
Change type of field docker.container.ip_addresses to
ip
instead ofkeyword
. 10364 - Rename http.request.body field to http.request.body.content. 10315
- Adjust php_fpm.process metricset fields to ECS. 10366
- Adjust mongodb.status metricset to to ECS. 10368
-
Refactor munin module to collect an event per plugin and to have more strict field mappings.
namespace
option has been removed, and will be replaced byservice.name
. 10322 - Change the following fields from type text to keyword: 10318
- ceph.osd_df.name
- ceph.osd_tree.name
- ceph.osd_tree.children
- kafka.consumergroup.meta
- kibana.stats.name
- mongodb.metrics.replication.executor.network_interface
- php_fpm.process.request_uri
- php_fpm.process.script
-
Add
service.name
option to all modules to explicitly setservice.name
if it is unset. 10427 - Update a few elasticsearch.* fields to map to ECS. 10350
- Update a few logstash.* fields to map to ECS. 10350
- Update a few kibana.* fields to map to ECS. 10350
- Update rabbitmq.* fields to map to ECS. 10563
- Update haproxy.* fields to map to ECS. 10558 10568
- Collect all EC2 meta data from all instances in all states. 10628
-
Fix MongoDB dashboard that had some incorrect field names from
status
Metricset 9795 9715
Packetbeat
Winlogbeat
- Adjust Winlogbeat fields to map to ECS. 10333
Functionbeat
Bugfixes
editAffecting all Beats
- Fix config appender registration. 9873
- Gracefully handle TLS options when enrolling a Beat. 9129
- The backing off now implements jitter to better distribute the load. 10172
- Fix TLS certificate DoS vulnerability. 10302
- Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289
- Fix encoding of timestamps when using disk spool. 10099
- Fix stopping of modules started by kubernetes autodiscover. 10476
- Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587
- Fix unauthorized error when loading dashboards by adding username and password into kibana config. 10513 10675
- Fix exclude_labels when there are dotted keys 10154
- Fix registry handle leak on Windows (https://github.com/elastic/go-sysinfo/pull/33). 9920
Auditbeat
- Enable System module config on Windows. 10237
Filebeat
- Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869, access log: 9955.
- Support haproxy log lines without captured headers. 9463 9958
- Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135
-
Fix bad bytes count in
docker
input when filtering by stream. 10211 -
Fixed data types for roles and indices fields in
elasticsearch/audit
fileset 10307 -
Ensure
source.address
is always populated by the nginx module (ECS). 10418 - Support mysql 5.7.22 slowlog starting with time information. 7892 9647
Heartbeat
Journalbeat
- Do not stop collecting events when journal entries change. 9994
Metricbeat
- Fix panics in vsphere module when certain values where not returned by the API. 9784
- Fix pod UID metadata enrichment in Kubernetes module. 10081
- Fix issue that would prevent collection of processes without command line on Windows. 10196
-
Fixed data type for tags field in
docker/container
metricset 10307 -
Fixed data type for tags field in
docker/image
metricset 10307 -
Fixed data type for isr field in
kafka/partition
metricset 10307 -
Fixed data types for various hosts fields in
mongodb/replstatus
metricset 10307 - Added function to close sql database connection. 10355
-
Fix issue with
elasticsearch/node_stats
metricset (x-pack) not indexingsource_node
field. 10639
Packetbeat
Winlogbeat
- Close handle on signalEvent. 9838
Functionbeat
Added
editAffecting all Beats
-
Update field definitions for
http
to ECS Beta 2 9645 -
Add
agent.id
andagent.ephemeral_id
fields to all beats. 9404 -
Add
name
config option toadd_host_metadata
processor. 9943 -
Add
add_labels
andadd_tags
processors. 9973 - Add missing file encoding to readers. 10080
-
Introduce
migration.enabled
configuration. 9805 - Add alias field support in Kibana index pattern. 10075
-
Add
add_fields
processor. 10119 - Add Kibana field formatter to bytes fields. 10184
-
Document a few more
auditd.log.*
fields. 10192 - Support Kafka 2.1.0. 10440
-
Add ILM mode
auto
to setup.ilm.enabled setting. This new default value detects if ILM is available 10347 - Add support to read ILM policy from external JSON file. 10347
-
Add
overwrite
andcheck_exists
settings to ILM support. 10347 - Generate Kibana index pattern on demand instead of using a local file. 10478
- Calls to Elasticsearch X-Pack APIs made by Beats won’t cause deprecation logs in Elasticsearch logs. {9656}9656[9656]
- Allow to unenroll a Beat from the UI. 9452
- Release Jolokia autodiscover as GA. 9706
- Allow Central Management to send events back to kibana. 9382
Auditbeat
- Add system module. 9546
-
Add
user.id
(UID) anduser.name
for ECS. 10195 -
Add
group.id
(GID) andgroup.name
for ECS. 10195 -
System module
process
dataset: Add user information to processes. 9963 -
Add system
package
dataset. 10225 -
Add system module
login
dataset. 9327 -
Add
entity_id
fields. 10500 - Add seven dashboards for the system module. 10511
Filebeat
-
Add
convert_timezone
option to Elasticsearch module to convert dates to UTC. 9756 9761 - Added module for parsing Google Santa logs. 9540
- Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399
- Add option to modules.yml file to indicate that a module has been moved 9432.
- Add support for ssl_request_log in apache2 module. 8088 9833
- Add support for iis 7.5 log format. 9753 9967
-
Add service.type field to all Modules. By default the field is set with the module name. It can be overwritten with
service.type
config. 10042 -
Add support for MariaDB in the
slowlog
fileset ofmysql
module. 9731 - Apache module’s error fileset now performs GeoIP lookup, like the access fileset. 10273
-
Elasticsearch module’s slowlog now populates
event.duration
(ECS). 9293 -
HAProxy module now populates
event.duration
andhttp.response.bytes
(ECS). 10143 - Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137
- Add convert_timezone to nginx module. 9839 10148
-
Add support for Percona in the
slowlog
fileset ofmysql
module. 6665 10227 - Added support for ingesting structured Elasticsearch audit logs 10352
- Added support for ingesting structured Elasticsearch slow logs 10445
- Added support for ingesting structured Elasticsearch deprecation logs 10445
- New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176
- Added support for ingesting structured Elasticsearch server logs 10428
- Populate more ECS fields in the Suricata module. 10006
- Add module zeek. 9931 10034
Heartbeat
-
Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the
docker
key. 10258
Journalbeat
- Migrate registry from previously incorrect path. 10486
Metricbeat
-
Add
key
metricset to the Redis module. 9582 9657 9746 -
Add
socket_summary
metricset to system defaults, removing experimental tag and supporting Windows 9709 -
Add docker
event
metricset. 9856 - Add performance metricset to x-pack mssql module 9826
- Add DeDot for kubernetes labels and annotations. 9860 9939
- Add more meaningful metrics to performance Metricset on MSSQL module 10011
-
Rename some fields in
performance
Metricset on MSSQL module to match the updated documentation from Microsoft 10074 - Add AWS EC2 module. 9257 9300
- Release windows Metricbeat module as GA. 10163
- Release traefik Metricbeat module as GA. 10166
- Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094
- List filesystems on Windows that have an access path but not an assigned letter 8916 10196
-
Add
nats
module. 10071 - Release uswgi Metricbeat module GA. 10164
- Release php_fpm module as GA. 10198
- Release Memcached module as GA. 10199
- Release etcd module as GA. 10200
- Release Ceph module as GA. 10202
- Release aerospike module as GA. 10203
- Release kubernetes apiserver and event metricsets as GA 10212
- Release Couchbase module as GA. 10201
- Release RabbitMQ module GA. 10165
- Release envoyproxy module GA. 10223
- Release mongodb.metrics and mongodb.replstatus as GA. 10242
- Release mysql.galera_status as GA. 10242
- Release postgresql.statement as GA. 10242
- Release RabbitMQ Metricbeat module GA. 10165
- Release Dropwizard module as GA. 10240
- Release Graphite module as GA. 10240
- Release kvm module as beta. 10279
- Release http.server metricset as GA. 10240
- Release Nats module as GA. 10281
- Release munin module as GA. 10311
- Release Golang module as GA. 10312
- Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222
- Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261
- Rename db Metricset to transaction_log in MSSQL Metricbeat module 10109
- Add process arguments and the path to its executable file in the system process metricset 10332
- Added server Metricset to Zookeeper Metricbeat module 8938 10341
- Release AWS module as GA. 10345
- Add overview dashboard to Zookeeper Metricbeat module 10379
Packetbeat
Functionbeat
- Mark Functionbeat as GA. 10564