Product release

The Beats 1.0.0

By
Monica Sarbu

When Packetbeat joined Elastic around 6 months ago, we had the vision of creating a framework of lightweight shippers, which we call “Beats” to capture all sorts of operational data from your servers and ship it to Logstash and Elasticsearch. Today we are excited to release the first production-ready version of the Beats platform, and the first officially supported Beats: Topbeat, Filebeat and Packetbeat.

We are doing a webinar about the Beats on December 8th. You can register here.


The Beats platform

Our goal was to build a platform that makes it easy for our community to create new Beats. For this we developed libbeat, the Go library that contains the common parts of all Beats for dealing with common tasks like inserting in bulk into Elasticsearch, securely sending events to Logstash, load-balancing the events to multiple Logstash and Elasticsearch nodes, and sending events in synchronous and asynchronous modes. The libbeat platform also includes mechanisms for detecting when downstream servers are getting overloaded or the network in between is getting congested, so it can reduce the sending rate.

beats platform.png

In other words, libbeat has everything you need to safely, securely, and efficiently send events to Logstash and Elasticsearch. Not only that, but it also takes care of things like configuration, CLI flags, and logging, so while creating a new Beat, you can just focus on capturing the data you are interested in. The rest of your analytics platform is handled by libbeat, Logstash, Elasticsearch, and Kibana.

We are very happy to see that besides our three official Beats, the community started to create their own, notably Dockerbeat, Pingbeat, UWSGIbeat and Nginxbeat. If this inspires you, there’s a developer guide for you.


Topbeat

We started Topbeat because we wanted a simple example Beat that can be used as a starting point by other Beats developers, but it turns out that a lot of people really like its simple value proposition. Think of it as a distributed top command on all your servers, and instead of printing the metrics on the screen, it sends them periodically to Logstash or Elasticsearch. This means you centralize system-wide data, such as system load, free/used memory, or disk usage, as well as per-process metrics (see complete list of metrics). No more SSH-ing around to see which processes are consuming the RAM.

Topbeat supports not only Linux, but also OS X and Windows. So you can install it on each server in your network and gather the same operating system metrics no matter which of the supported operating systems you use.

After you’ve installed Topbeat on your server, it starts periodically sending metrics to a central point in Elasticsearch. If you are running Kibana on top of Elasticsearch, you are able to visualize your data and configure your own customizable dashboards by using widgets like: System load, Servers overview, System-wide memory usage or cpu usage, Top processes, Per-process memory usage or cpu usage, Disk usage overview, and Disk usage.

Screen Shot 2015-11-22 at 11.41.24 PM.png


Filebeat

Filebeat is the successor of the Logstash Forwarder, a lightweight log shipper that has been used in production by many companies for years. Logstash-Forwarder is a simple lightweight Go application that forwards all the logs of your servers to a central location for further processing.

Logstash-Forwarder was started by the creator of Logstash, Jordan Sissel and maintained by the Logstash developers. Unfortunately it tended to lag behind in terms of improvements and bug fixes when compared to Logstash itself. Because of the clear similarities with the Beats, we decided the best path forward was to transform the Logstash Forwarder into a Beat. So we took the Forwarder code, we split it into pieces, replaced the rusty parts, added unit tests, and then put it all back together into Filebeat.

Configuring Filebeat is simple. After installing it on your servers, just configure the paths for Filebeat to crawl and it will start sending your logs to Elasticsearch via Logstash for further processing. It is intelligent enough to deal with log rotation, file renames, and the temporary unavailability of the downstream server, so you never lose a log line.

If you are a Logstash Forwarder user, please follow the migration guide to get started with Filebeat.

Screen Shot 2015-11-23 at 11.03.28 AM.png


Packetbeat

Packetbeat is the first Beat and the one that started it all. Imagine you would like to monitor and troubleshoot a complex distributed system where each component is using different programming languages, web frameworks, or database technologies. What these components have in common is that the communication between them is done over the network, typically using standard protocols like HTTP for REST API calls.

Think of Packetbeat as a distributed, real-time Wireshark that correlates the request with the response in a transaction and inserts data about each transaction into Elasticsearch.

Packetbeat gives you visibility into your application by looking at the communication between the servers in your network. Because it works by passively sniffing the network traffic, it doesn’t interfere with your application.

The power of Packetbeat lies in its open source nature: anyone can add support for new protocols, even proprietary ones. In the last few months, the community added support for MongoDB, DNS over UDP, and DNS over TCP and there a few more community contributions currently in progress. We are always happy to help contributors get their code into Packetbeat, and we have recently written a developer guide for new protocols.

packetbeat-statistics.png


The path to GA

Do you want to get the whole story of the 1.0.0 release? Check out the previous blog posts about intermediate releases:


A big Thank You to everyone who contributed code, reported issues, or just tried the Beats during the beta stage. Start your experience with Beats 1.0.0 and let us know what you think on discuss, Twitter, or open an issue on GitHub.