Beats and Elastic Agent capabilities
editBeats and Elastic Agent capabilities
editElastic provides two main ways to send data to Elasticsearch:
- Beats are lightweight data shippers that send operational data to Elasticsearch. Elastic provides separate Beats for different types of data, such as logs, metrics, and uptime. Depending on what data you want to collect, you may need to install multiple shippers on a single host.
-
Elastic Agent is a single agent for logs, metrics, security data, and threat prevention. The Elastic Agent can be deployed in two different modes:
- Managed by Fleet — easily deploy services with the Fleet UI. Once installed, the Elastic Agent lifecycle and policy/configuration is managed from a central point.
- Standalone mode — once installed, all configuration is applied to the Elastic Agent manually. See Run Elastic Agent standalone (advanced users) for more information.
Running Elastic Agent in standalone mode is an advanced use case. The documentation is incomplete and not yet mature. When possible, we recommend using Fleet-managed agents instead of standalone mode.
The method you use depends on your use case, which features you need, and whether you want to centrally manage your agents.
Beats and Elastic Agent can both send data directly to Elasticsearch or via Logstash, where you can further process and enhance the data, before visualizing it in Kibana.
This article summarizes the features and functionality you need to be aware of before adding new Elastic Agents or replacing your current Beats with Elastic Agents. Starting in version 7.14.0, Elastic Agent is generally available (GA).
Choosing between Elastic Agent and Beats
editElastic Agent is a single binary designed to provide the same functionality that the various Beats provide today. However, some functionality gaps are being addressed as we strive to reach feature parity.
The following steps will help you determine if Elastic Agent can support your use case:
-
Determine if the integrations you need are supported and Generally Available (GA) on Elastic Agent. Integrations that are currently GA are:
-
Auditd -
Elastic Endpoint Security -
Fortinet -
Netflow -
Office365 -
Okta -
Palo Alto Networks -
Suricata -
System -
Windows -
ZeekFor more details, refer to Integrations.
-
- If the integration is available, check Supported outputs to see whether the required output is also supported.
- Review Capabilities comparison to determine if any features required by your deployment are supported. Elastic Agent should support most of the features available on Beats and is updated for each release.
If you are satisfied with all three steps, then Elastic Agent is suitable for your deployment. However, if any steps fail your assessment, you should continue using Beats, and review future updates or contact us in the discuss forum.
Supported inputs
editFor Elastic Agents that are centrally managed by Fleet, data collection is further simplified and defined by integrations. In this model, the decision on the inputs is driven by the integration you want to collect data from. The complexity of configuration details of various inputs is driven centrally by Fleet and specifically by the integrations.
The following table shows the inputs supported by the Elastic Agent in 7.15.2:
| Input | Beats | Fleet-managed Elastic Agent | Standalone Elastic Agent |
|---|---|---|---|
AWS CloudWatch |
|
|
|
AWS S3 |
|
|
|
Azure Event Hub |
|
|
|
Cloud Foundry |
|
|
|
Container |
|
|
|
Docker |
|
|
|
Endpoint security |
|
|
|
GCP Pub/Sub |
|
|
|
HTTP Endpoint |
|
|
|
HTTP JSON |
|
|
|
Kafka |
|
|
|
Logfile |
|
|
|
MQTT |
|
|
|
NetFlow |
|
|
|
O365 Mgmt Act API |
|
|
|
Osquery |
|
|
|
Redis |
|
|
|
Stdin |
|
|
|
Syslog |
|
|
|
TCP |
|
|
|
UDP |
|
|
|
UNIX |
|
|
|
Supported outputs
editThe following table shows the outputs supported by the Elastic Agent in 7.15.2:
Endpoint Security has a different output matrix.
| Output | Beats | Fleet-managed Elastic Agent | Standalone Elastic Agent |
|---|---|---|---|
Elasticsearch Service |
|
|
|
Elasticsearch |
|
|
|
Logstash |
|
Under consideration |
|
Kafka |
|
Under consideration |
Under consideration |
Redis |
|
|
|
File |
|
|
|
Console |
|
|
|
Currently, Elastic Agents managed by Fleet can only output to the same Elasticsearch cluster where Fleet is running. Support for outputting to remote Elasticsearch clusters is under consideration for a future release.
Supported configurations
edit| Beats configuration | Elastic Agent support |
|---|---|
Supported via integrations. |
|
Not configurable. Set to default values. |
|
Many of these global settings are now internal to the agent and for proper operations should not be modified. |
|
Elastic Agent configures these paths to provide a simpler and more streamlined configuration experience. |
|
Config is distributed via policy. |
|
Related to the config file reload. |
|
Configured through Fleet. See Supported outputs. |
|
Supported |
|
Enabled by default although the Agent uses data streams. |
|
No longer applicable |
|
New Elastic Agent workflow doesn’t need this. |
|
New Elastic Agent workflow doesn’t need this. |
|
Processors can be defined at the integration level. |
|
Autodiscover is facilitated through dynamic inputs. Elastic Agent does not support hints-based autodiscovery. |
|
Elastic Agent does not expose the internal memory queues to the end user. You can configure output queue parameters to tune your environment, and the Agent takes care of configuring the internal queues to accomplish your tuning intent. |
|
Within the Fleet UI, you can add yaml settings to configure multiple hosts per output type, which enables loadbalancing. |
|
Supported |
|
Supported |
|
Supported |
Capabilities comparison
editThe following table shows a comparison of capabilities supported by Beats and the Elastic Agent in 7.15.2:
| Item | Beats | Fleet-managed Elastic Agent | Standalone Elastic Agent | Description |
|---|---|---|---|---|
Single agent for all use cases |
|
|
|
Elastic Agent provides logs, metrics, and more. You’d need to install multiple Beats for these use cases. |
Install integrations from web UI or API |
|
|
|
Elastic Agent integrations are installed with a convenient web UI or API, but Beats modules are installed with a CLI. This installs Elasticsearch assets such as index templates and ingest pipelines, and Kibana assets such as dashboards. |
Configure from web UI or API |
|
|
|
Fleet-managed Elastic Agent integrations can be configured in the web UI or API. Standalone Elastic Agent can use the web UI, API, or YAML. Beats can only be configured via YAML files. |
Central, remote agent policy management |
|
|
|
Elastic Agent policies can be centrally managed through Fleet. You have to manage Beats configuration yourself or with a third-party solution. |
Central, remote agent binary upgrades |
|
|
|
Elastic Agents can be remotely upgraded through Fleet. You have to upgrade Beats yourself or with a third-party solution. |
Add Kibana and Elasticsearch assets for a single integration or module |
|
|
|
Elastic Agent integrations allow you to add Kibana and Elasticsearch assets for a single integration at a time. Beats installs hundreds of assets for all modules at once. |
Auto-generated Elasticsearch API keys |
|
|
|
Fleet can automatically generate API keys with limited permissions for each Elastic Agent, which can be individually revoked. Standalone Elastic Agent and Beats require you to create and manage credentials, and users often share them across hosts. |
Auto-generate minimal Elasticsearch permissions |
|
|
|
Fleet can automatically give Elastic Agents minimal output permissions based on the inputs running. With standalone Elastic Agent and Beats, users often give overly broad permissions because it’s more convenient. |
Data streams support |
|
|
|
Elastic Agents use data streams with easier index life cycle management and the data stream naming scheme. Beats uses a single index with potentially thousands of fields. |
Variables and input conditions |
|
|
|
Elastic Agent offers variables and input conditions to dynamically adjust based on the local host environment. Users can configure these directly in YAML for standalone Elastic Agent or using the Fleet API for Fleet-managed Elastic Agent. The Integrations app allows users to enter variables, and we are considering a UI to edit conditions. Beats only offers static configuration. |
Allow non-superusers to manage assets and agents |
|
|
|
We require a superuser role to use the Fleet and Integrations apps and corresponding APIs. We are considering changing this requirement. These apps are optional for standalone Elastic Agent. Beats offers finer grained roles. |
Air-gapped network support |
|
|
|
The Integrations app requires a network connection to the Elastic Package Registry. We are considering an on-prem version of EPR. Fleet-managed Elastic Agents require a connection to our artifact repository for agent binary upgrades. These are not required for standalone Elastic Agents or Beats. |
Run without root on host |
|
|
|
Fleet-managed Elastic Agents require root permission, in particular for Endpoint Security. Standalone Elastic Agents and Beats do not. |
Multiple outputs |
|
|
|
Fleet-managed Elastic Agents only provide a single global output to the same Elasticsearch cluster where Fleet is running. We are considering support for more outputs. |
Separate monitoring cluster |
|
|
|
Fleet-managed Elastic Agents only provide a single global output to the same Elasticsearch cluster where Fleet is running. We are considering support for remote monitoring clusters. Standalone Elastic Agent and Beats can send to a remote monitoring cluster. |
Secret management |
|
|
|
Elastic Agent stores credentials in the agent policy. We are considering adding keystore support. Beats allows users to access credentials in a local keystore. |
Progressive or canary deployments |
|
|
|
Fleet does not have a feature to deploy an Elastic Agent policy update progressively but we are considering improved support. With standalone Elastic Agent and Beats you can deploy configuration files progressively using third party solutions. |
Multiple configurations per host |
|
|
|
Elastic Agent uses a single Elastic Agent policy for configuration, and uses variables and input conditions to adapt on a per-host basis. Beats supports multiple configuration files per host, enabling third party solutions to deploy files hierarchically or in multiple groups, and enabling finer-grained access control to those files. |
Compatible with version control and infrastructure as code solutions |
|
|
|
Fleet stores the agent policy in Elasticsearch. It does not integrate with external version control or infrastructure as code solutions, but we are considering improved support. However, Beats and Elastic Agent in standalone mode use a YAML file that is compatible with these solutions. |
Elastic Agent monitoring support
editYou configure the collection of agent metrics in the agent policy. If metrics collection is selected (the default), all Elastic Agents enrolled in the policy will send metrics data to Elasticsearch (the output is configured globally).
The following image shows the Agent monitoring settings for the default agent policy:
There are also pre-built dashboards for agent metrics that you can access under Assets in the Elastic Agent integration:
The [Elastic Agent] Agent metrics dashboard shows an aggregated view of agent metrics:
