Introducing Beats Central Management in the Elastic Stack

By
Matt Apperson
•
Alvin Chen

In the world of data analytics, everything starts with data collection. This means that data needs to be ingested quickly, reliably, and it must be structured appropriately so it’s available immediately for analytical use. The ability to do exactly that, regardless of the size, shape, or speed of your data set, is why the Beats product suite is now the core data shipper for many popular use cases. And with many thanks to our community, it continues to expand as we build more off-the-shelf solutions like modules and the Infrastructure UI. With this growth of overall adoption and per deployment footprint, it has become increasingly important for users and enterprises to securely achieve operational excellence, especially when running Beats at scale.

By nature, Beats are often deployed on the edge where deployments can easily span into the thousands. With this tier of scale, monitoring and management become two key pillars of operational excellence. In 6.2, we released Beats monitoring to provide deep visibility into overall deployment health and status. Most recently in 6.5, we are proud to introduce Beats central management which provides a centralized Kibana UI for managing and updating Beats configurations. This feature aims to drastically reduce users’ reliance on external configuration management tools like Ansible, Puppet, or Chef, enabling Beats configuration changes to be managed directly with the Elastic Stack (namely Kibana and Elasticsearch) and rolled out automatically to entire fleets of Beats. Have you heard that modules can now be managed centrally too?

Beats Central Management

Beats centralized management is available in 6.5 as a beta feature. With security posture top-of-mind, this feature is available under the Elastic Gold license, or Standard license when using our Elastic Cloud service, to ensure a properly secured deployment. It comes inclusive of a new Beats central management UI in Kibana and leverages Elasticsearch as a centralized configuration store. In the near future, we also plan to expose an API for easier integration with external tools and systems.


The new interface

Core to the central management experience, is the new Beats central management UI. Found in the Management tab of Kibana, this new interface allows users to easily and securely enroll Beats into the central management workflow. After enrollment, these Beats and their configurations can then be managed and automatically orchestrated directly through the interface, regardless of whether it's one Beats or thousands.

New enroll command on Filebeat and Metricbeat

To enable control of a Beat to central management, Filebeat and Metricbeat (with others to follow) now have a new enroll command that is used to enroll your Beat with a token generated in our UI, or via a username and password of a restricted use user. This therefore allows easy and simple integration regardless of your workflow!

Beats, now with Configuration Tags

Managing the intricacies of a large scale deployment can be challenging. Thus for central management we introduced the concept of tags for Beats configurations. Tags are a collection of separate chunks of configuration data, allowing you to share configuration between Beats in whole or in part. We felt this concept will help to keep things both organized and flexible to multiple use cases.

Security

Anytime you introduce a management tool, there is the potential for additional attack vectors. Thus we have taken additional security steps to ensure your infrastructure is safe when using Beats central management, such as:

  • Unique authentication tokens per Beat, ensuring that one compromised edge node will not compromise your stack
  • Additional user roles to allow control over who can access central management
  • Auto-expiring enrollment tokens
  • And of course, Elasticsearch authentication and authorization mechanisms

Feedback Welcome (and Encouraged)

The ability to monitor and manage your Beats deployments through a single pane of Kibana glass is an experience that’s worthwhile to explore and we hope you give it a spin. Although the Beats central management feature is currently in beta, we’re actively working towards making it GA production ready. In the meantime, we’re curious to hear your feedback (please submit a Github issue, all our code is open!) on the feature as well as what other Beats (including Community Beats) you’d like to see supported next. Check out the Beats central management documentation to get started!