Overview

Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to each host. A single agent makes it easier and faster to deploy monitoring across your infrastructure. The agent’s single, unified policy makes it easier to add integrations for new data sources.

Fleet provides a web-based UI in Kibana to add and manage integrations for popular services and platforms, as well as manage a fleet of Elastic Agents. Our integrations provide an easy way to add new sources of data, plus they ship with out-of-the-box assets like dashboards, visualizations, and pipelines to extract structured fields out of logs. This makes it easier to get insights within seconds.

Fleet provides a web-based UI for configuring integrations with your data sources. This includes popular services and platforms like Nginx or AWS, as well as many generic input types like log files.

The Elastic Agent policy allows you to use any number of integrations for data sources. You can apply the Elastic Agent policy to multiple agents, making it even easier to manage configuration at scale.

When you add an integration, you select the agent policy to use then configure inputs for logs and metrics, such as the path to your Nginx access logs. When you’re done, you save the integration to update the Elastic Agent policy. The next time enrolled agents check in, they receive the update. Having the policies automatically deployed is more convenient than doing it yourself by using SSH, Ansible playbooks, or some other tool.

If you prefer infrastructure as code, you may use YAML files and APIs. Fleet has an API-first design. Anything you can do in the UI, you can also do using the API. This makes it easy to automate and integrate with other systems.

You can see the state of all your Elastic Agents in Fleet. On the Agents page, you can see which agents are online, which have errors, and the last time they checked in. You can also see the version of the Elastic Agent binary and policy.

Fleet serves as the communication channel back to the Elastic Agents. Agents check in for the latest updates on a regular basis. You can have any number of agents enrolled into each agent policy, which allows you to scale up to thousands of hosts. When you make a change to an agent policy, all the agents receive the update during their next check-in. You no longer have to distribute policy updates yourself.

The data collected by Elastic Agent is stored in indices that are more granular than you’d get by default with Filebeat. This gives you more visibility into the sources of data volume, and control over lifecycle management policies and index permissions. These indices are called data streams.

As you can see in the following screen, each data stream (or index) is broken out by dataset, type, and namespace.

The dataset is defined by the integration and describes the fields and other settings for each index. For example, you might have one dataset for process metrics with a field describing whether the process is running or not, and another dataset for disk I/O metrics with a field describing the number of bytes read.

This indexing strategy solves the issue of having indices with hundreds or thousands of fields. Because each index stores only a small number of fields, the indices are more compact with faster autocomplete. And as an added bonus, the Discover page only shows relevant fields.

Namespaces are user-defined strings that allow you to group data any way you like. A namespace can be up to 100 bytes in length (multi-byte characters will count toward this limit faster). For example, you might group your data by environment (prod, QA) or by team name. Using a namespace makes it easier to search the data from a given source by using index patterns, or to give users permissions to data by assigning an index pattern to user roles.

When searching your data in Kibana, you can use an index pattern to search across all or some of the indices.