Beats version 7.0.0
editBeats version 7.0.0
editThe list below covers the changes during the 7.0.0-alpha1, -alpha2, -beta1, -rc1 and -rc2 releases.
Also read Breaking changes for more detail about changes that affect upgrade.
Breaking changes
editAffecting all Beats
-
Empty
meta.json
file will be treated as a missing meta file. 8558 - Removed dashboards and index patterns generation for Kibana 5. 8927
- On systems with systemd, the Beats log is now written to journald by default rather than file. To revert this behaviour override BEAT_LOG_OPTS with an empty value. 8942.
- Automatically cap signed integers to 63 bits. 8991
- Use _doc as document type. 9056
- Update add_cloud_metadata fields to adjust to ECS. 9265
- Rename beat.timezone to event.timezone. 9458
- Embedded html is not escaped anymore by default. 9914
- Remove port settings from Logstash and Redis output. 9934
-
Rename
process.exe
toprocess.executable
in add_process_metadata to align with ECS. 9949 - Remove --configtest command line flag. 10138
- Remove --setup command line flag. 10138
- Remove --version command line flag. 10138
-
Import ECS change ecs#308:
leaf field
user.group
is now thegroup
field set. 10275 - Docker and Kubernetes labels/annotations will be "dedoted" by default. 10338
- ILM will be available by default if Elasticsearch > 7.0 is used. 10347
- Move output.elasticsearch.ilm settings to setup.ilm. 10347
- On Google Cloud Engine (GCE) the add_cloud_metadata will now trim the project info from the cloud.machine.type and cloud.availability_zone. 10968
-
Rename
migration.enabled
config tomigration.6_to_7.enabled
. 11284
Auditbeat
- Rename beat.name to agent.type, beat.hostname to agent.hostname, beat.version to agent.version.
-
Use
initial_scan
action for new paths. 7954 -
Remove warning for deprecated option:
filters
. 9002 -
Rename
source.hostname
tosource.domain
in the auditd module. 9027 -
Rename
process.exe
toprocess.executable
in auditd module to align with ECS. 9949 -
Rename
process.cwd
toprocess.working_directory
in auditd module to align with ECS. 10195 -
Change data type of
process.pid
andprocess.ppid
to number in JSON output of the auditd module. 10195 -
Change data type of
file.uid
andfile.gid
to string in JSON output of the FIM module. 10195 - Rename user fields to ECS in auditd module. 10456
-
Rename
event.type
toauditd.message_type
in auditd module because event.type is reserved for future use by ECS. 10536 -
Field
file.origin
changed type fromtext
tokeyword
. 10544 -
Rename
auditd.messages
toevent.original
andauditd.warnings
toerror.message
. 10577 - Process dataset: Only report processes with executable. 11232
- Shorten entity IDs. 11405
Filebeat
-
Rename
fileset.name
toevent.name
. 8879 -
Rename
fileset.module
toevent.module
. 8879 -
Rename
source
tolog.file.path
andlog.source.ip
. 8902 -
Remove the deprecated
prospectors
option in the configuration. Useinputs
instead. 8909 -
Rename
offset
tolog.offset
. 8923 - Modify apache/error dataset to follow ECS. 8963
-
Rename
source_ecs
tosource
in the Filebeat Suricata module. 8983 -
Remove warnings for deprecated options:
spool_size
,publish_async
,idle_timeout
. 9002 -
Rename many
traefik.access.*
fields to map to ECS. 9005 -
Rename many
nginx.access.*
fields to map to ECS. 9081 -
Rename many
iis.access.*
fields to map to ECS. 9084 -
IIS module’s user agent string is no longer encoded (
+
replaced with spaces). 9084 -
Rename many
haproxy.*
fields to map to ECS. 9117 -
Rename many
system.syslog.*
fields to map to ECS. 9135 -
Rename many
system.auth.*
fields to map to ECS. 9138 -
Rename many
apache2.access.*
fields to map to ECS. 9245 -
Rename a few
elasticsearch.audit.*
fields to map to ECS. 9293 -
Rename many
kibana.log.*
fields to map to ECS. 9301 -
Rename
apache2
module toapache
. 9402 - Fix parsing of GC entries in elasticsearch server log. 9513 9810
-
Rename
read_timestamp
toevent.created
for Redis input. 9924 -
Rename a few
logstash.*
fields to map to ECS. Removelogstash.slowlog.message
. 9935 -
Rename many
iis.error.*
fields to map to ECS. 9955 -
Rename a few
nginx.error.*
fields to map to ECS. 10007 -
Rename a few
mysql.*
fields to map to ECS. 10008 -
Rename a few
mongodb.*
fields to map to ECS. 10009 -
Remove
service.name
from Elastcsearch module. Replace withservice.type
. 10042 -
Rename
read_timestamp
toevent.created
for all Filebeat modules using it. 10139 -
Now save the first seen timestamp in
event.created
(previouslyread_timestamp
), instead of saving the parsed date. Now aligned withevent.created
semantics elsewhere. 10139 -
Adjust fileset
haproxy.log
to map to ECS. 10143 -
Rename
mysql.error.thread_id
andmysql.slowlog.id
tomysql.thread_id
. 10161 -
Remove
mysql.error.timestamp
andmysql.slowlog.timestamp
. 10161 -
Rename multiple fields to
http.response.body.bytes
, from modules "apache", "iis", "kibana", "nginx" and "traefik", includinghttp.response.content_length
(ECS). 10188 -
Rename many
auditd.log.*
fields to map to ECS. 10192 -
Remove numeric coercions for
user.id
andgroup.id
. IDs should bekeyword
. 10233 -
Migrate multiple fields to
event.duration
, from modules "apache", "elasticsearch", "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik", includinghttp.response.elapsed_time
(ECS). 10188, 10274 - Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above. 10352
- Migrate Elasticsearch audit logs fields to ECS. 10352
-
Change type of
haproxy.log
fileset fields from text to keyword:response.captured_headers
,request.captured_headers
,raw_request_line
,mode
. 10397 -
Remove field
kafka.log.trace.full
fromkafka.log
fileset. 10398 -
Change field
kafka.log.class
forkafka.log
fileset from text to keyword. 10398 -
Change type of field
backend_url
andfrontend_name
intraefik.access
metricset to type keyword. 10401 -
Several text fields in the Elasticsearch module are now indexed as
keyword
fields withtext
multi-fields (ECS). 10414 -
Several text fields in the Logstash module are now indexed as
keyword
fields withtext
multi-fields (ECS). 10417 -
Move dissect pattern for
traefik.access
fileset from Filbeat to Elasticsearch. 10442 -
The
elasticsearch/deprecation
fileset now indexes thecomponent
field underelasticsearch
instead ofelasticsearch.server
. 10445 -
Rename setting
filebeat.registry_flush
tofilebeat.registry.flush
. 10504 -
Rename setting
filebeat.registry_file_permission
tofilebeat.registry.file_permission
. 10504 -
Remove setting
filebeat.registry_file
in favor offilebeat.registry.path
. The registry file will be stored in a sub-directory now. 10504 -
Address
add_kubernetes_metadata
processor issue where old source field is still used for matcher. 10505 10506 -
Change type of
haproxy.source
from text to keyword. 10506 -
Rename
event.type
tosuricata.eve.event_type
in Suricata module becauseevent.type
is reserved for future use by ECS. 10575 -
Set
ecs: true
inuser_agent
processors when loading pipelines with Filebeat 7.0.x into Elasticsearch 6.7.x. 10655 10875
Heartbeat
-
A number of fields have been aliased to their relevant counterparts in the
url.*
field. Existing visualizations should mostly work. The fields that have been moved aremonitor.scheme -> url.scheme
,monitor.host -> url.domain
,resolve.host -> url.domain
,http.url -> url.full
,tcp.port -> url.port
. In addition to these moves the new fieldsurl.username
,url.password
,url.path
, andurl.query
are now present. It should be noted that theurl.password
field does not contain actual password values, but rather the text<hidden>
9570. -
Monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values.
To have continuity with the old format of monitor IDs, set the
id
property explicitly. 9697 - The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. 10294
Journalbeat
Metricbeat
-
event.duration
is now in nano and not microseconds anymore. 8941 -
Remove warning for deprecated option:
filters
. 9002 - Refactor Prometheus metric mappings. 9948
- Remove Prometheus stats metricset in favor of just using Prometheus collector. 9948
-
Rename
http.request.body
field tohttp.request.body.content
. 10315 - Change the following fields from type text to keyword: 10318
-
ceph.osd_df.name
-
ceph.osd_tree.name
-
ceph.osd_tree.children
-
kafka.consumergroup.meta
-
kibana.stats.name
-
mongodb.metrics.replication.executor.network_interface
-
php_fpm.process.request_uri
-
php_fpm.process.script
-
Adjust
redis.info
metricset fields to ECS. 10319 -
Refactor munin module to collect an event per plugin and to have more strict field mappings.
The
namespace
option has been removed and will be replaced byservice.name
. 10322 - Migrate system process metricset fields to ECS. 10332
- Migrate system socket metricset fields to ECS. 10339
- Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. 10339
-
Update a few
elasticsearch.* fields
to map to ECS. 10350 -
Update a few
kibana.*
fields to map to ECS. 10350 -
Update a few
logstash.*
fields to map to ECS. 10350 -
Change type of field
docker.container.ip_addresses
toip
instead ofkeyword
. 10364 -
Adjust
php_fpm.process
metricset fields to ECS. 10366 -
Adjust
mongodb.status
metricset to to ECS. 10368 -
Add
service.name
option to all modules to explicitly setservice.name
if it is unset. 10427 -
Update
rabbitmq.*
fields to map to ECS. 10563 -
Update
haproxy.*
fields to map to ECS. 10558 10568 - Collect all EC2 metadata from all instances in all states. 10628
- Migrate docker module to ECS. 10927
- Add connection and request timeouts for HTTP helper. 11032
Packetbeat
- Change Packetbeat fields to align with ECS. 7968
- Rename the flow event fields to follow ECS. 9121
- Rename several client and server fields. IP, port, and process metadata are now contained under the client and server namespaces. 9303
-
Adjust Packetbeat
http
fields to ECS. 9645 -
http.request.body
moves tohttp.request.body.content
-
http.response.body
moves tohttp.response.body.content
- Remove trailing dot from domain names reported by the DNS protocol. 9941
Winlogbeat
- Adjust Winlogbeat fields to map to ECS. 10333
Bugfixes
editAffecting all Beats
-
Fix support of
add_docker_metadata
in Windows by identifying systems' path separator. 7797 -
Fix
-d
CLI flag by trimming spaces from selectors. 7864 - Start autodiscover consumers before producers. 7926
-
Fix
exclude_labels
when there are dotted keys. 10154 - Fix unauthorized error when loading dashboards by adding username and password into kibana config. 10513 10675
- Allow to configure Kafka fetching strategy for the topic metadata. 10682
- Reconnections of Kubernetes watchers are now logged at debug level when they are harmless. 10988
-
Add
missing host.*
fields to fields.yml. 11016 -
Fixed OS family classification in
add_host_metadata
for Amazon Linux, Raspbian, and RedHat Linux. 9134 11494 - Relax validation of the X-Pack license UID value. 11640
- Fix a parsing error with the X-Pack license check on 32-bit system. 11650
Filebeat
-
Rename many
icinga.*
fields to map to ECS. 9294 -
Rename many
kafka.log.*
fields to map to ECS. 9297 -
Rename many
postgresql.log.*
fields to map to ECS. 9308 -
Rename many
redis.log.*
fields to map to ECS. 9315 -
Use
log.source.address
instead oflog.source.ip
for network input sources. 9487 - Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869, access log: 9955.
-
Ensure
source.address
is always populated by the nginx module (ECS). 10418 -
Fix errors in filebeat Zeek dashboard and README files. Add
notice.log
support. 10916 - Fix a bug when converting NetFlow fields to snake_case. 10950
-
Add
on_failure
handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test case. 11004 11105 - Fix issue preventing docker container events to be stored if the container has a network interface without ip address. 11225 11247
- Fix goroutine leak happening when harvesters are dynamically stopped. 11263
- Don’t apply multiline rules in Logstash json logs. 11346
-
Fix panic in
add_kubernetes_metadata
processor when keylog
does not exist. 11543 11549
Heartbeat
Metricbeat
- Fix for not reusable http client leading to connection leaks in Jolokia module. 11014
- Collect metrics when EC2 instances are not in running state. 11008 11023
-
Change ECS field
cloud.provider
toaws
. 11023 -
Fix
ec2
metricset to collect metrics from Cloudwatch with the same timestamp. 11142 -
Add missing
aws.ec2.instance.state.name
into fields.yml. 11219 11221 - Fix potential memory leak in stopped docker metricsets. 11294
Packetbeat
- Fixed the mysql missing transactions if monitoring a connection from the start. 8173
Winlogbeat
- Close handle on signalEvent. 9838
Added
editAffecting all Beats
-
Add field
host.os.kernel
to theadd_host_metadata
processor and to the internal monitoring data. 7807 - Add debug check to logp.Logger 7965
- Count HTTP 429 responses in the elasticsearch output. 8056
- Allow Bus to buffer events in case listeners are not configured. 8527
-
Perform
add_cloud_metadata
initialization asynchronously to avoid delays on startup. 8845 -
Autodiscovery no longer requires that the
condition
field be set. If left unset all configs will be matched. 9029 -
Add geo fields to
add_host_metadata
processor. 9392 -
Add
agent.id
andagent.ephemeral_id
fields to all beats. 9404 -
Add dedot method in
add_docker_metadata
processor in libbeat. 9350 9505 -
Update field definitions for
http
to ECS. 9645 - Calls to Elasticsearch X-Pack APIs made by Beats won’t cause deprecation logs in Elasticsearch logs. 9656
-
Introduce
migration.enabled
configuration. 9805 -
Add
name
config option toadd_host_metadata
processor. 9943 -
Add
add_labels
andadd_tags
processors. 9973 - Add alias field support in Kibana index pattern. 10075
- Add missing file encoding to readers. 10080
-
Add
add_fields
processor. 10119 - Add Kibana field formatter to bytes fields. 10184
-
Add ILM mode
auto
tosetup.ilm.enabled
setting. This new default value detects if ILM is available 10347 - Add support to read ILM policy from external JSON file. 10347
-
Add
overwrite
andcheck_exists
settings to ILM support. 10347 - Support Kafka 2.1.0. 10440
- Generate Kibana index pattern on demand instead of using a local file. 10478
Auditbeat
Filebeat
- Add custom unpack to log hints config to avoid env resolution. 7710
- Make docker input check if container strings are empty. 7960
-
Keep unparsed user agent information in
user_agent.original
. 8537 -
Elasticsearch module’s slowlog now populates
event.duration
(ECS). 9293 - Add option to modules.yml file to indicate that a module has been moved. 9432.
- Added module for parsing Google Santa logs. 9540
- Add module zeek. 9931 10034
-
Add
service.type
field to all Modules. By default the field is set with the module name. It can be overwritten withservice.type
config. 10042 -
HAProxy module now populates
event.duration
andhttp.response.bytes
(ECS). 10143 -
Apache module’s
error
fileset now performs GeoIP lookup, like theaccess
fileset. 10273 - Added support for ingesting structured Elasticsearch audit logs. 10352
- Added support for ingesting structured Elasticsearch server logs. 10428
- Added support for ingesting structured Elasticsearch deprecation logs. 10445
- Added support for ingesting structured Elasticsearch slow logs. 10445
- Add ISO8601 timestamp support in syslog metricset. 8716 10736
- Add support for loading custom NetFlow and IPFIX field definitions to netflow input. 10945 11223
- Added categorization fields for SSH login events in the system/auth fileset. 11334
- Add support for MySQL 8.0, Percona 8.0 and MariaDB 10.3. 11417
Heartbeat
- Add central management support. 9254
Metricbeat
- Add metrics about cache size to memcached module. 7740
-
Add
service.type
field to Metricbeat. 8965 - Add AWS EC2 module. 9257 9300
- Add MS SQL module to X-Pack. 9414
-
Add
socket_summary
metricset to system defaults. Remove experimental tag and support Windows. 9709 -
Add
key
metricset to the Redis module. 9582 9657 9746 -
Add
performance
metricset to X-Pack mssql module. 9826 -
Add more meaningful metrics to
performance
metricset in MSSQL module. 10011 -
Add
nats
module. 10071 -
Rename some fields in
performance
metricset on MSSQL module to match the updated documentation from Microsoft. 10074 -
Rename
db
metricset totransaction_log
in MSSQL Metricbeat module. 10109 - Release Kvm module as beta. 10279
- Release Nats module as GA. 10281
- Release Munin module as GA. 10311
- Release Golang module as GA. 10312
- Add process arguments and the path to its executable file in the system process metricset. 10332
- Release AWS module as GA. 10345
- Add filters and pie chart for AWS EC2 dashboard. 10596
Packetbeat
-
Add support to decode HTTP bodies compressed with
gzip
anddeflate
. 7915 - Add support to decode mysql prepared statement command. 8084
- Added support to calculate certificates' fingerprints (MD5, SHA-1, SHA-256). 8180
-
Add
network.community_id
to Packetbeat flow events. 10061 - Add aliases for flow fields that were renamed. 7968 10063
Known Issue
editJournalbeat
- Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).