Beats highlights

This list summarizes the most important enhancements in Beats. For the complete list, go to Beats release highlights.

The Elastic Common Schema, or ECS, is an open source specification that defines a common set of document fields for event data ingested into Elasticsearch. ECS makes it dramatically easier for users to correlate data across sources and develop common content, such as dashboards and machine learning jobs.

In 7.0, all Beats and Beats modules generate ECS format events by default. This means that adopting ECS is as easy as upgrading to Beats 7.0. All Beats module dashboards in 7.0 make use of ECS.

Migrating to a common schema means that many fields have been renamed. We have developed an upgrade procedure that uses Elasticsearch field aliases to make the transition easier. After the upgrade is complete, we strongly advise that you adjust your custom Kibana dashboards, machine learning jobs, and other content to use the new ECS field names.

See the Beats upgrade documentation for more information.

In 6.6, Elasticsearch added advanced capabilities for index management. Rather than simply performing management actions on your indices on a set schedule, you can base actions on other factors such as shard size and performance requirements. You control how indices are handled as they age by attaching a lifecycle policy to the index template used to create them. You can update the policy to modify the lifecycle of both new and existing indices. This set of capabilities are grouped in the index lifecycle management (ILM) APIs.

In 7.0, Beats defaults to rotating indices by using ILM policies, if the Elasticsearch version to which they connect supports ILM. The default policy rotates indices when they reach 50 GB or 30 days. You can edit the ILM policy by using the Kibana management UI, or directly via the Elasticsearch API.

The full suite of modules to monitor your Elastic Stack are now GA. These include the Metricbeat modules for Elasticsearch, Logstash, and Kibana.

In the future, we will switch to Metricbeat as the recommended agent for monitoring the Elastic Stack. To prepare for the switch, see Collecting Elasticsearch monitoring data with Metricbeat.

Beats adds several new modules, focusing on datastores and the cloud.

On the cloud side, Metricbeat adds the AWS module, which collects and centralizes basic resource utilization metrics from all your EC2 instances, directly from Cloudwatch. A widely used messaging platform, NATS, earns its own module for capturing stats, connections, routes, and subscriptions metrics.

For datastores, Metricbeat offers modules for Microsoft SQL Server and CouchDB. The MSSQL module captures transaction log and performance counters, while the CouchDB module provides a server metricset.

For data relevant to security analytics, Filebeat adds a Zeek module that integrates with the popular open-source Zeek project, formerly known as Bro, and a Santa module, which tracks process executions on macOS. These modules add to the list of data sources already supported in the 6.x series, including Suricata, IPtables, and NetFlow.

In addition, the Auditbeat system module keeps improving, and the transition to ECS makes all Beats modules more useful for security use cases.