IMPORTANT: No additional bug fixes or documentation updates
will be released for this version. For the latest information, see the
current release documentation.
Beats version 7.0.0-beta1
editBeats version 7.0.0-beta1
editBreaking changes
editAffecting all Beats
- Embedded html is not escaped anymore by default. 9914
- Remove port settings from Logstash and Redis output. 9934
-
Rename
process.exe
toprocess.executable
in add_process_metadata to align with ECS. 9949 -
Import ECS change ecs#308:
leaf field
user.group
is now thegroup
field set. 10275 - Update the code of Central Management to align with the new returned format. 10019
- Docker and Kubernetes labels/annotations will be "dedoted" by default. 10338
- Remove --setup command line flag. 10138
- Remove --version command line flag. 10138
- Remove --configtest command line flag. 10138
- Move output.elasticsearch.ilm settings to setup.ilm. 10347
- ILM will be available by default if Elasticsearch > 7.0 is used. 10347
Auditbeat
-
Rename
process.exe
toprocess.executable
in auditd module to align with ECS. 9949 -
Rename
process.cwd
toprocess.working_directory
in auditd module to align with ECS. 10195 -
Change data type of
process.pid
andprocess.ppid
to number in JSON output of the auditd module. 10195 -
Change data type of
file.uid
andfile.gid
to string in JSON output of the FIM module. 10195 -
Field
file.origin
changed type fromtext
tokeyword
. 10544 - Rename user fields to ECS in auditd module. 10456
-
Rename
event.type
toauditd.message_type
in auditd module because event.type is reserved for future use by ECS. 10536 -
Rename
auditd.messages
toevent.original
andauditd.warnings
toerror.message
. 10577
Filebeat
-
Rename many
kibana.log.*
fields to map to ECS. 9301 - Modify apache/error dataset to follow ECS. 8963
-
Rename many
traefik.access.*
fields to map to ECS. 9005 - Fix parsing of GC entries in elasticsearch server log. 9513 9810
-
Rename
read_timestamp
toevent.created
for Redis input. 9924 -
Rename a few
elasticsearch.audit.*
fields to map to ECS. 9293 -
Rename
read_timestamp
toevent.created
for all Filebeat modules using it. 10139 -
Rename many
iis.error.*
fields to map to ECS. 9955 -
Adjust fileset
haproxy.log
to map to ECS. 10143 -
Rename a few
logstash.*
fields to map to ECS, remove logstash.slowlog.message. 9935 -
Rename a few
mongodb.*
fields to map to ECS. 10009 -
Rename a few
mysql.*
fields to map to ECS. 10008 -
Rename a few
nginx.error.*
fields to map to ECS. 10007 -
Rename many
auditd.log.*
fields to map to ECS. 10192 - Filesets with multiple ingest pipelines added in 8914 only work with Elasticsearch >= 6.5.0 10001
- Remove service.name from Elastcsearch module. Replace by service.type. 10042
-
Remove numeric coercions for
user.id
andgroup.id
. IDs should bekeyword
. 10233 - Add grok pattern to support redis 5.0.3 log timestamp. 9819 10033
-
Now save the first seen timestamp in
event.created
(previouslyread_timestamp
), instead of saving the parsed date. Now aligned withevent.created
semantics elsewhere. 10139 -
Rename
mysql.error.thread_id
andmysql.slowlog.id
tomysql.thread_id
. 10161 -
Remove
mysql.error.timestamp
andmysql.slowlog.timestamp
. 10161 -
Migrate multiple fields to
event.duration
, from modules "apache", "elasticsearch", "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik", includinghttp.response.elapsed_time
(ECS). 10188, 10274 -
Rename multiple fields to
http.response.body.bytes
, from modules "apache", "iis", "kibana", "nginx" and "traefik", includinghttp.response.content_length
(ECS). 10188 -
Change type from haproxy.log fileset fields from text to keyword: response.captured_headers, request.captured_headers,
raw_request_line
,mode
. 10397 - Change type of field backend_url and frontend_name in traefik.access metricset to type keyword. 10401
- Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above 10352
- Migrate Elasticsearch audit logs fields to ECS 10352
-
Several text fields in the Logstash module are now indexed as
keyword
fields withtext
multi-fields (ECS). 10417 -
Several text fields in the Elasticsearch module are now indexed as
keyword
fields withtext
multi-fields (ECS). 10414 - Move dissect pattern for traefik.access fileset from Filbeat to Elasticsearch. 10442
-
The
elasticsearch/deprecation
fileset now indexes thecomponent
field underelasticsearch
instead ofelasticsearch.server
. 10445 -
Remove field
kafka.log.trace.full
from kafka.log fielset. 10398 -
Change field
kafka.log.class
for kafka.log fileset from text to keyword. 10398 - Address add_kubernetes_metadata processor issue where old source field is still used for matcher. 10505 10506
- Change type of haproxy.source from text to keyword. 10506
-
Rename
event.type
tosuricata.eve.event_type
in Suricata module because event.type is reserved for future use by ECS. 10575 - Populate more ECS fields in the Suricata module. 10006
-
Rename setting
filebeat.registry_flush
tofilebeat.registry.flush
. 10504 -
Rename setting
filebeat.registry_file_permission
tofilebeat.registry.file_permission
. 10504 -
Remove setting
filebeat.registry_file
in favor offilebeat.registry.path
. The registry file will be stored in a sub-directory by now. 10504
Heartbeat
- Remove monitor generator script that was rarely used. 9648
-
monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values. If you wish to have continuity with the old format of monitor IDs you’ll need to set the
id
property explicitly. 9697 -
A number of fields have been aliased to their relevant counterparts in the
url.*
field. Existing visualizations should mostly work. The fields that have been moved aremonitor.scheme -> url.scheme
,monitor.host -> url.domain
,resolve.host -> url.domain
,http.url -> url.full
,tcp.port -> url.port
. In addition to these moves the new fieldsurl.username
,url.password
,url.path
, andurl.query
are now present. It should be noted that theurl.password
field does not contain actual password values, but rather the text<hidden>
9570. - The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. 10294
Journalbeat
Metricbeat
- Migrate system process metricset fields to ECS. 10332
- Refactor Prometheus metric mappings 9948
- Removed Prometheus stats metricset in favor of just using Prometheus collector 9948
- Migrate system socket metricset fields to ECS. 10339
- Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. 10339
- Adjust Redis.info metricset fields to ECS. 10319
-
Change type of field docker.container.ip_addresses to
ip
instead ofkeyword
. 10364 - Rename http.request.body field to http.request.body.content. 10315
- Adjust php_fpm.process metricset fields to ECS. 10366
- Adjust mongodb.status metricset to to ECS. 10368
-
Refactor munin module to collect an event per plugin and to have more strict field mappings.
namespace
option has been removed, and will be replaced byservice.name
. 10322 - Change the following fields from type text to keyword: 10318
- ceph.osd_df.name
- ceph.osd_tree.name
- ceph.osd_tree.children
- kafka.consumergroup.meta
- kibana.stats.name
- mongodb.metrics.replication.executor.network_interface
- php_fpm.process.request_uri
- php_fpm.process.script
-
Add
service.name
option to all modules to explicitly setservice.name
if it is unset. 10427 - Update a few elasticsearch.* fields to map to ECS. 10350
- Update a few logstash.* fields to map to ECS. 10350
- Update a few kibana.* fields to map to ECS. 10350
- Update rabbitmq.* fields to map to ECS. 10563
- Update haproxy.* fields to map to ECS. 10558 10568
- Collect all EC2 meta data from all instances in all states. 10628
-
Fix MongoDB dashboard that had some incorrect field names from
status
Metricset 9795 9715
Packetbeat
Winlogbeat
- Adjust Winlogbeat fields to map to ECS. 10333
Functionbeat
Bugfixes
editAffecting all Beats
- Fix config appender registration. 9873
- Gracefully handle TLS options when enrolling a Beat. 9129
- The backing off now implements jitter to better distribute the load. 10172
- Fix TLS certificate DoS vulnerability. 10302
- Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289
- Fix encoding of timestamps when using disk spool. 10099
- Fix stopping of modules started by kubernetes autodiscover. 10476
- Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587
- Fix unauthorized error when loading dashboards by adding username and password into kibana config. 10513 10675
- Fix exclude_labels when there are dotted keys 10154
- Fix registry handle leak on Windows (https://github.com/elastic/go-sysinfo/pull/33). 9920
Auditbeat
- Enable System module config on Windows. 10237
Filebeat
- Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869, access log: 9955.
- Support haproxy log lines without captured headers. 9463 9958
- Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135
-
Fix bad bytes count in
docker
input when filtering by stream. 10211 -
Fixed data types for roles and indices fields in
elasticsearch/audit
fileset 10307 -
Ensure
source.address
is always populated by the nginx module (ECS). 10418 - Support mysql 5.7.22 slowlog starting with time information. 7892 9647
Heartbeat
Journalbeat
- Do not stop collecting events when journal entries change. 9994
Metricbeat
- Fix panics in vsphere module when certain values where not returned by the API. 9784
- Fix pod UID metadata enrichment in Kubernetes module. 10081
- Fix issue that would prevent collection of processes without command line on Windows. 10196
-
Fixed data type for tags field in
docker/container
metricset 10307 -
Fixed data type for tags field in
docker/image
metricset 10307 -
Fixed data type for isr field in
kafka/partition
metricset 10307 -
Fixed data types for various hosts fields in
mongodb/replstatus
metricset 10307 - Added function to close sql database connection. 10355
-
Fix issue with
elasticsearch/node_stats
metricset (x-pack) not indexingsource_node
field. 10639
Packetbeat
Winlogbeat
- Close handle on signalEvent. 9838
Functionbeat
Added
editAffecting all Beats
-
Update field definitions for
http
to ECS Beta 2 9645 -
Add
agent.id
andagent.ephemeral_id
fields to all beats. 9404 -
Add
name
config option toadd_host_metadata
processor. 9943 -
Add
add_labels
andadd_tags
processors. 9973 - Add missing file encoding to readers. 10080
-
Introduce
migration.enabled
configuration. 9805 - Add alias field support in Kibana index pattern. 10075
-
Add
add_fields
processor. 10119 - Add Kibana field formatter to bytes fields. 10184
-
Document a few more
auditd.log.*
fields. 10192 - Support Kafka 2.1.0. 10440
-
Add ILM mode
auto
to setup.ilm.enabled setting. This new default value detects if ILM is available 10347 - Add support to read ILM policy from external JSON file. 10347
-
Add
overwrite
andcheck_exists
settings to ILM support. 10347 - Generate Kibana index pattern on demand instead of using a local file. 10478
- Calls to Elasticsearch X-Pack APIs made by Beats won’t cause deprecation logs in Elasticsearch logs. {9656}9656[9656]
- Allow to unenroll a Beat from the UI. 9452
- Release Jolokia autodiscover as GA. 9706
- Allow Central Management to send events back to kibana. 9382
Auditbeat
- Add system module. 9546
-
Add
user.id
(UID) anduser.name
for ECS. 10195 -
Add
group.id
(GID) andgroup.name
for ECS. 10195 -
System module
process
dataset: Add user information to processes. 9963 -
Add system
package
dataset. 10225 -
Add system module
login
dataset. 9327 -
Add
entity_id
fields. 10500 - Add seven dashboards for the system module. 10511
Filebeat
-
Add
convert_timezone
option to Elasticsearch module to convert dates to UTC. 9756 9761 - Added module for parsing Google Santa logs. 9540
- Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399
- Add option to modules.yml file to indicate that a module has been moved 9432.
- Add support for ssl_request_log in apache2 module. 8088 9833
- Add support for iis 7.5 log format. 9753 9967
-
Add service.type field to all Modules. By default the field is set with the module name. It can be overwritten with
service.type
config. 10042 -
Add support for MariaDB in the
slowlog
fileset ofmysql
module. 9731 - Apache module’s error fileset now performs GeoIP lookup, like the access fileset. 10273
-
Elasticsearch module’s slowlog now populates
event.duration
(ECS). 9293 -
HAProxy module now populates
event.duration
andhttp.response.bytes
(ECS). 10143 - Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137
- Add convert_timezone to nginx module. 9839 10148
-
Add support for Percona in the
slowlog
fileset ofmysql
module. 6665 10227 - Added support for ingesting structured Elasticsearch audit logs 10352
- Added support for ingesting structured Elasticsearch slow logs 10445
- Added support for ingesting structured Elasticsearch deprecation logs 10445
- New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176
- Added support for ingesting structured Elasticsearch server logs 10428
- Populate more ECS fields in the Suricata module. 10006
- Add module zeek. 9931 10034
Heartbeat
-
Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the
docker
key. 10258
Journalbeat
- Migrate registry from previously incorrect path. 10486
Metricbeat
-
Add
key
metricset to the Redis module. 9582 9657 9746 -
Add
socket_summary
metricset to system defaults, removing experimental tag and supporting Windows 9709 -
Add docker
event
metricset. 9856 - Add performance metricset to x-pack mssql module 9826
- Add DeDot for kubernetes labels and annotations. 9860 9939
- Add more meaningful metrics to performance Metricset on MSSQL module 10011
-
Rename some fields in
performance
Metricset on MSSQL module to match the updated documentation from Microsoft 10074 - Add AWS EC2 module. 9257 9300
- Release windows Metricbeat module as GA. 10163
- Release traefik Metricbeat module as GA. 10166
- Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094
- List filesystems on Windows that have an access path but not an assigned letter 8916 10196
-
Add
nats
module. 10071 - Release uswgi Metricbeat module GA. 10164
- Release php_fpm module as GA. 10198
- Release Memcached module as GA. 10199
- Release etcd module as GA. 10200
- Release Ceph module as GA. 10202
- Release aerospike module as GA. 10203
- Release kubernetes apiserver and event metricsets as GA 10212
- Release Couchbase module as GA. 10201
- Release RabbitMQ module GA. 10165
- Release envoyproxy module GA. 10223
- Release mongodb.metrics and mongodb.replstatus as GA. 10242
- Release mysql.galera_status as GA. 10242
- Release postgresql.statement as GA. 10242
- Release RabbitMQ Metricbeat module GA. 10165
- Release Dropwizard module as GA. 10240
- Release Graphite module as GA. 10240
- Release kvm module as beta. 10279
- Release http.server metricset as GA. 10240
- Release Nats module as GA. 10281
- Release munin module as GA. 10311
- Release Golang module as GA. 10312
- Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222
- Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261
- Rename db Metricset to transaction_log in MSSQL Metricbeat module 10109
- Add process arguments and the path to its executable file in the system process metricset 10332
- Added server Metricset to Zookeeper Metricbeat module 8938 10341
- Release AWS module as GA. 10345
- Add overview dashboard to Zookeeper Metricbeat module 10379
Packetbeat
Functionbeat
- Mark Functionbeat as GA. 10564