IMPORTANT: No additional bug fixes or documentation updates
will be released for this version. For the latest information, see the
current release documentation.
Beats version 7.0.0
editBeats version 7.0.0
editThe list below covers the changes during the 7.0.0-alpha1, -alpha2, -beta1, -rc1 and -rc2 releases.
Also read Breaking changes in 7.0 for more detail about changes that affect upgrade.
Breaking changes
editAffecting all Beats
-
Empty
meta.json
file will be treated as a missing meta file. 8558 - Removed dashboards and index patterns generation for Kibana 5. 8927
- On systems with systemd, the Beats log is now written to journald by default rather than file. To revert this behaviour override BEAT_LOG_OPTS with an empty value. 8942.
- Automatically cap signed integers to 63 bits. 8991
- Use _doc as document type. 9056
- Update add_cloud_metadata fields to adjust to ECS. 9265
- Rename beat.timezone to event.timezone. 9458
- Embedded html is not escaped anymore by default. 9914
- Remove port settings from Logstash and Redis output. 9934
-
Rename
process.exe
toprocess.executable
in add_process_metadata to align with ECS. 9949 - Remove --configtest command line flag. 10138
- Remove --setup command line flag. 10138
- Remove --version command line flag. 10138
-
Import ECS change ecs#308:
leaf field
user.group
is now thegroup
field set. 10275 - Docker and Kubernetes labels/annotations will be "dedoted" by default. 10338
- ILM will be available by default if Elasticsearch > 7.0 is used. 10347
- Move output.elasticsearch.ilm settings to setup.ilm. 10347
- On Google Cloud Engine (GCE) the add_cloud_metadata will now trim the project info from the cloud.machine.type and cloud.availability_zone. 10968
-
Rename
migration.enabled
config tomigration.6_to_7.enabled
. 11284
Auditbeat
- Rename beat.name to agent.type, beat.hostname to agent.hostname, beat.version to agent.version.
-
Use
initial_scan
action for new paths. 7954 -
Remove warning for deprecated option:
filters
. 9002 -
Rename
source.hostname
tosource.domain
in the auditd module. 9027 -
Rename
process.exe
toprocess.executable
in auditd module to align with ECS. 9949 -
Rename
process.cwd
toprocess.working_directory
in auditd module to align with ECS. 10195 -
Change data type of
process.pid
andprocess.ppid
to number in JSON output of the auditd module. 10195 -
Change data type of
file.uid
andfile.gid
to string in JSON output of the FIM module. 10195 - Rename user fields to ECS in auditd module. 10456
-
Rename
event.type
toauditd.message_type
in auditd module because event.type is reserved for future use by ECS. 10536 -
Field
file.origin
changed type fromtext
tokeyword
. 10544 -
Rename
auditd.messages
toevent.original
andauditd.warnings
toerror.message
. 10577 - Process dataset: Only report processes with executable. 11232
- Shorten entity IDs. 11405
Filebeat
-
Rename
fileset.name
toevent.name
. 8879 -
Rename
fileset.module
toevent.module
. 8879 -
Rename
source
tolog.file.path
andlog.source.ip
. 8902 -
Remove the deprecated
prospectors
option in the configuration. Useinputs
instead. 8909 -
Rename
offset
tolog.offset
. 8923 - Modify apache/error dataset to follow ECS. 8963
-
Rename
source_ecs
tosource
in the Filebeat Suricata module. 8983 -
Remove warnings for deprecated options:
spool_size
,publish_async
,idle_timeout
. 9002 -
Rename many
traefik.access.*
fields to map to ECS. 9005 -
Rename many
nginx.access.*
fields to map to ECS. 9081 -
Rename many
iis.access.*
fields to map to ECS. 9084 -
IIS module’s user agent string is no longer encoded (
+
replaced with spaces). 9084 -
Rename many
haproxy.*
fields to map to ECS. 9117 -
Rename many
system.syslog.*
fields to map to ECS. 9135 -
Rename many
system.auth.*
fields to map to ECS. 9138 -
Rename many
apache2.access.*
fields to map to ECS. 9245 -
Rename a few
elasticsearch.audit.*
fields to map to ECS. 9293 -
Rename many
kibana.log.*
fields to map to ECS. 9301 -
Rename
apache2
module toapache
. 9402 - Fix parsing of GC entries in elasticsearch server log. 9513 9810
-
Rename
read_timestamp
toevent.created
for Redis input. 9924 -
Rename a few
logstash.*
fields to map to ECS. Removelogstash.slowlog.message
. 9935 -
Rename many
iis.error.*
fields to map to ECS. 9955 -
Rename a few
nginx.error.*
fields to map to ECS. 10007 -
Rename a few
mysql.*
fields to map to ECS. 10008 -
Rename a few
mongodb.*
fields to map to ECS. 10009 -
Remove
service.name
from Elastcsearch module. Replace withservice.type
. 10042 -
Rename
read_timestamp
toevent.created
for all Filebeat modules using it. 10139 -
Now save the first seen timestamp in
event.created
(previouslyread_timestamp
), instead of saving the parsed date. Now aligned withevent.created
semantics elsewhere. 10139 -
Adjust fileset
haproxy.log
to map to ECS. 10143 -
Rename
mysql.error.thread_id
andmysql.slowlog.id
tomysql.thread_id
. 10161 -
Remove
mysql.error.timestamp
andmysql.slowlog.timestamp
. 10161 -
Rename multiple fields to
http.response.body.bytes
, from modules "apache", "iis", "kibana", "nginx" and "traefik", includinghttp.response.content_length
(ECS). 10188 -
Rename many
auditd.log.*
fields to map to ECS. 10192 -
Remove numeric coercions for
user.id
andgroup.id
. IDs should bekeyword
. 10233 -
Migrate multiple fields to
event.duration
, from modules "apache", "elasticsearch", "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik", includinghttp.response.elapsed_time
(ECS). 10188, 10274 - Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above. 10352
- Migrate Elasticsearch audit logs fields to ECS. 10352
-
Change type of
haproxy.log
fileset fields from text to keyword:response.captured_headers
,request.captured_headers
,raw_request_line
,mode
. 10397 -
Remove field
kafka.log.trace.full
fromkafka.log
fileset. 10398 -
Change field
kafka.log.class
forkafka.log
fileset from text to keyword. 10398 -
Change type of field
backend_url
andfrontend_name
intraefik.access
metricset to type keyword. 10401 -
Several text fields in the Elasticsearch module are now indexed as
keyword
fields withtext
multi-fields (ECS). 10414 -
Several text fields in the Logstash module are now indexed as
keyword
fields withtext
multi-fields (ECS). 10417 -
Move dissect pattern for
traefik.access
fileset from Filbeat to Elasticsearch. 10442 -
The
elasticsearch/deprecation
fileset now indexes thecomponent
field underelasticsearch
instead ofelasticsearch.server
. 10445 -
Rename setting
filebeat.registry_flush
tofilebeat.registry.flush
. 10504 -
Rename setting
filebeat.registry_file_permission
tofilebeat.registry.file_permission
. 10504 -
Remove setting
filebeat.registry_file
in favor offilebeat.registry.path
. The registry file will be stored in a sub-directory now. 10504 -
Address
add_kubernetes_metadata
processor issue where old source field is still used for matcher. 10505 10506 -
Change type of
haproxy.source
from text to keyword. 10506 -
Rename
event.type
tosuricata.eve.event_type
in Suricata module becauseevent.type
is reserved for future use by ECS. 10575 -
Set
ecs: true
inuser_agent
processors when loading pipelines with Filebeat 7.0.x into Elasticsearch 6.7.x. 10655 10875
Heartbeat
-
A number of fields have been aliased to their relevant counterparts in the
url.*
field. Existing visualizations should mostly work. The fields that have been moved aremonitor.scheme -> url.scheme
,monitor.host -> url.domain
,resolve.host -> url.domain
,http.url -> url.full
,tcp.port -> url.port
. In addition to these moves the new fieldsurl.username
,url.password
,url.path
, andurl.query
are now present. It should be noted that theurl.password
field does not contain actual password values, but rather the text<hidden>
9570. -
Monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values.
To have continuity with the old format of monitor IDs, set the
id
property explicitly. 9697 - The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. 10294
Journalbeat
Metricbeat
-
event.duration
is now in nano and not microseconds anymore. 8941 -
Remove warning for deprecated option:
filters
. 9002 - Refactor Prometheus metric mappings. 9948
- Remove Prometheus stats metricset in favor of just using Prometheus collector. 9948
-
Rename
http.request.body
field tohttp.request.body.content
. 10315 - Change the following fields from type text to keyword: 10318
-
ceph.osd_df.name
-
ceph.osd_tree.name
-
ceph.osd_tree.children
-
kafka.consumergroup.meta
-
kibana.stats.name
-
mongodb.metrics.replication.executor.network_interface
-
php_fpm.process.request_uri
-
php_fpm.process.script
-
Adjust
redis.info
metricset fields to ECS. 10319 -
Refactor munin module to collect an event per plugin and to have more strict field mappings.
The
namespace
option has been removed and will be replaced byservice.name
. 10322 - Migrate system process metricset fields to ECS. 10332
- Migrate system socket metricset fields to ECS. 10339
- Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. 10339
-
Update a few
elasticsearch.* fields
to map to ECS. 10350 -
Update a few
kibana.*
fields to map to ECS. 10350 -
Update a few
logstash.*
fields to map to ECS. 10350 -
Change type of field
docker.container.ip_addresses
toip
instead ofkeyword
. 10364 -
Adjust
php_fpm.process
metricset fields to ECS. 10366 -
Adjust
mongodb.status
metricset to to ECS. 10368 -
Add
service.name
option to all modules to explicitly setservice.name
if it is unset. 10427 -
Update
rabbitmq.*
fields to map to ECS. 10563 -
Update
haproxy.*
fields to map to ECS. 10558 10568 - Collect all EC2 metadata from all instances in all states. 10628
- Migrate docker module to ECS. 10927
- Add connection and request timeouts for HTTP helper. 11032
Packetbeat
- Change Packetbeat fields to align with ECS. 7968
- Rename the flow event fields to follow ECS. 9121
- Rename several client and server fields. IP, port, and process metadata are now contained under the client and server namespaces. 9303
-
Adjust Packetbeat
http
fields to ECS. 9645 -
http.request.body
moves tohttp.request.body.content
-
http.response.body
moves tohttp.response.body.content
- Remove trailing dot from domain names reported by the DNS protocol. 9941
Winlogbeat
- Adjust Winlogbeat fields to map to ECS. 10333
Bugfixes
editAffecting all Beats
-
Fix support of
add_docker_metadata
in Windows by identifying systems' path separator. 7797 -
Fix
-d
CLI flag by trimming spaces from selectors. 7864 - Start autodiscover consumers before producers. 7926
-
Fix
exclude_labels
when there are dotted keys. 10154 - Fix unauthorized error when loading dashboards by adding username and password into kibana config. 10513 10675
- Allow to configure Kafka fetching strategy for the topic metadata. 10682
- Reconnections of Kubernetes watchers are now logged at debug level when they are harmless. 10988
-
Add
missing host.*
fields to fields.yml. 11016 -
Fixed OS family classification in
add_host_metadata
for Amazon Linux, Raspbian, and RedHat Linux. 9134 11494 - Relax validation of the X-Pack license UID value. 11640
- Fix a parsing error with the X-Pack license check on 32-bit system. 11650
Filebeat
-
Rename many
icinga.*
fields to map to ECS. 9294 -
Rename many
kafka.log.*
fields to map to ECS. 9297 -
Rename many
postgresql.log.*
fields to map to ECS. 9308 -
Rename many
redis.log.*
fields to map to ECS. 9315 -
Use
log.source.address
instead oflog.source.ip
for network input sources. 9487 - Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869, access log: 9955.
-
Ensure
source.address
is always populated by the nginx module (ECS). 10418 -
Fix errors in filebeat Zeek dashboard and README files. Add
notice.log
support. 10916 - Fix a bug when converting NetFlow fields to snake_case. 10950
-
Add
on_failure
handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test case. 11004 11105 - Fix issue preventing docker container events to be stored if the container has a network interface without ip address. 11225 11247
- Fix goroutine leak happening when harvesters are dynamically stopped. 11263
- Don’t apply multiline rules in Logstash json logs. 11346
-
Fix panic in
add_kubernetes_metadata
processor when keylog
does not exist. 11543 11549
Heartbeat
Metricbeat
- Fix for not reusable http client leading to connection leaks in Jolokia module. 11014
- Collect metrics when EC2 instances are not in running state. 11008 11023
-
Change ECS field
cloud.provider
toaws
. 11023 -
Fix
ec2
metricset to collect metrics from Cloudwatch with the same timestamp. 11142 -
Add missing
aws.ec2.instance.state.name
into fields.yml. 11219 11221 - Fix potential memory leak in stopped docker metricsets. 11294
Packetbeat
- Fixed the mysql missing transactions if monitoring a connection from the start. 8173
Winlogbeat
- Close handle on signalEvent. 9838
Added
editAffecting all Beats
-
Add field
host.os.kernel
to theadd_host_metadata
processor and to the internal monitoring data. 7807 - Add debug check to logp.Logger 7965
- Count HTTP 429 responses in the elasticsearch output. 8056
- Allow Bus to buffer events in case listeners are not configured. 8527
-
Perform
add_cloud_metadata
initialization asynchronously to avoid delays on startup. 8845 -
Autodiscovery no longer requires that the
condition
field be set. If left unset all configs will be matched. 9029 -
Add geo fields to
add_host_metadata
processor. 9392 -
Add
agent.id
andagent.ephemeral_id
fields to all beats. 9404 -
Add dedot method in
add_docker_metadata
processor in libbeat. 9350 9505 -
Update field definitions for
http
to ECS. 9645 - Calls to Elasticsearch X-Pack APIs made by Beats won’t cause deprecation logs in Elasticsearch logs. 9656
-
Introduce
migration.enabled
configuration. 9805 -
Add
name
config option toadd_host_metadata
processor. 9943 -
Add
add_labels
andadd_tags
processors. 9973 - Add alias field support in Kibana index pattern. 10075
- Add missing file encoding to readers. 10080
-
Add
add_fields
processor. 10119 - Add Kibana field formatter to bytes fields. 10184
-
Add ILM mode
auto
tosetup.ilm.enabled
setting. This new default value detects if ILM is available 10347 - Add support to read ILM policy from external JSON file. 10347
-
Add
overwrite
andcheck_exists
settings to ILM support. 10347 - Support Kafka 2.1.0. 10440
- Generate Kibana index pattern on demand instead of using a local file. 10478
Auditbeat
Filebeat
- Add custom unpack to log hints config to avoid env resolution. 7710
- Make docker input check if container strings are empty. 7960
-
Keep unparsed user agent information in
user_agent.original
. 8537 -
Elasticsearch module’s slowlog now populates
event.duration
(ECS). 9293 - Add option to modules.yml file to indicate that a module has been moved. 9432.
- Added module for parsing Google Santa logs. 9540
- Add module zeek. 9931 10034
-
Add
service.type
field to all Modules. By default the field is set with the module name. It can be overwritten withservice.type
config. 10042 -
HAProxy module now populates
event.duration
andhttp.response.bytes
(ECS). 10143 -
Apache module’s
error
fileset now performs GeoIP lookup, like theaccess
fileset. 10273 - Added support for ingesting structured Elasticsearch audit logs. 10352
- Added support for ingesting structured Elasticsearch server logs. 10428
- Added support for ingesting structured Elasticsearch deprecation logs. 10445
- Added support for ingesting structured Elasticsearch slow logs. 10445
- Add ISO8601 timestamp support in syslog metricset. 8716 10736
- Add support for loading custom NetFlow and IPFIX field definitions to netflow input. 10945 11223
- Added categorization fields for SSH login events in the system/auth fileset. 11334
- Add support for MySQL 8.0, Percona 8.0 and MariaDB 10.3. 11417
Heartbeat
- Add central management support. 9254
Metricbeat
- Add metrics about cache size to memcached module. 7740
-
Add
service.type
field to Metricbeat. 8965 - Add AWS EC2 module. 9257 9300
- Add MS SQL module to X-Pack. 9414
-
Add
socket_summary
metricset to system defaults. Remove experimental tag and support Windows. 9709 -
Add
key
metricset to the Redis module. 9582 9657 9746 -
Add
performance
metricset to X-Pack mssql module. 9826 -
Add more meaningful metrics to
performance
metricset in MSSQL module. 10011 -
Add
nats
module. 10071 -
Rename some fields in
performance
metricset on MSSQL module to match the updated documentation from Microsoft. 10074 -
Rename
db
metricset totransaction_log
in MSSQL Metricbeat module. 10109 - Release Kvm module as beta. 10279
- Release Nats module as GA. 10281
- Release Munin module as GA. 10311
- Release Golang module as GA. 10312
- Add process arguments and the path to its executable file in the system process metricset. 10332
- Release AWS module as GA. 10345
- Add filters and pie chart for AWS EC2 dashboard. 10596
Packetbeat
-
Add support to decode HTTP bodies compressed with
gzip
anddeflate
. 7915 - Add support to decode mysql prepared statement command. 8084
- Added support to calculate certificates' fingerprints (MD5, SHA-1, SHA-256). 8180
-
Add
network.community_id
to Packetbeat flow events. 10061 - Add aliases for flow fields that were renamed. 7968 10063
Known Issue
editJournalbeat
- Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).