IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Deploy Elastic Agent to send data Ingest application traces with Elastic Agent »

› › ›

Ingest logs, metrics, and uptime data with Elastic Agent

edit

Ingest logs, metrics, and uptime data with Elastic Agent

edit

This guide describes how to:

Monitor logs and metrics from systems and services across your organization
Monitor the availability of your HTTP, TCP, and ICMP services using the Synthetics integration
Monitor Nginx logs and metrics using the Nginx integration

For feedback and questions, please contact us in the discuss forum.

Prerequisites

edit

You need Elasticsearch for storing and searching your data, and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud (recommended), or self-manage the Elastic Stack on your own hardware.

Here’s what you need for each deployment type:

Elasticsearch Service deployment that includes an Integrations Server (included by default in every Elasticsearch Service deployment). Our hosted Elasticsearch Service is available on AWS, GCP, and Azure, and you can try it for free.
Kibana user with All privileges on Fleet and Integrations. Since many Integrations assets are shared across spaces, users need the Kibana privileges in all spaces.

Elasticsearch cluster and Kibana (version 8.3) with a basic license or higher. Learn how to install the Elastic Stack on your own hardware.
Secure, encrypted connection between Kibana and Elasticsearch. For more information, see Start the Elastic Stack with security enabled.
Internet connection for Kibana to download integration packages from the Elastic Package Registry. Make sure the Kibana server can connect to https://epr.elastic.co on port 443. If your environment has network traffic restrictions, there are ways to work around this requirement. See Air-gapped environments for more information.
Kibana user with All privileges on Fleet and Integrations. Since many Integrations assets are shared across spaces, users need the Kibana privileges in all spaces.
In the Elasticsearch configuration, the built-in API key service must be enabled. (xpack.security.authc.api_key.enabled: true)
In the Kibana configuration, the saved objects encryption key must be set. Fleet requires this setting in order to save API keys and encrypt them in Kibana. You can either set xpack.encryptedSavedObjects.encryptionKey to an alphanumeric value of at least 32 characters, or run the kibana-encryption-keys command to generate the key.

Example security settings

For testing purposes, you can use the following settings to get started quickly, but make sure you properly secure the Elastic Stack before sending real data.

elasticsearch.yml example:

xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true

kibana.yml example:

elasticsearch.username: "kibana_system" 
xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters"

The password should be stored in the Kibana keystore as described in the Elasticsearch security documentation.

Step 1: Set up Fleet

edit

Use Fleet in Kibana to get logs, metrics, and security data into the Elastic Stack.

Not using Fleet? Advanced users who want to configure and manage Elastic Agents manually can run agents standalone.

The first time you use Fleet, you might need to set it up and add a Fleet Server:

Elastic Cloud runs a hosted version of Integrations Server that includes Fleet Server. No extra setup is required unless you want to scale your deployment.

To confirm that an Integrations Server is available in your deployment:

In Kibana, go to Management > Fleet.
On the Agents tab, look for the Elastic Cloud agent policy. This policy is managed by Elastic Cloud, and contains a Fleet Server integration and an Elastic APM integration. You cannot modify the policy. Confirm that the agent status is Healthy.

Don’t see the agent? Make sure your deployment includes an Integrations Server instance. This instance is required to use Fleet.

To deploy a self-managed Fleet Server, you install an Elastic Agent and enroll it in an agent policy containing the Fleet Server integration.

You can install only a single Elastic Agent per host, which means you cannot run Fleet Server and another Elastic Agent on the same host unless you deploy a containerized Fleet Server.

In Kibana, go to Management > Fleet > Settings. For more information about these settings, see Fleet settings.
Under Fleet Server hosts, click Edit hosts and specify one or more host URLs your Elastic Agents will use to connect to Fleet Server. For example, https://192.0.2.1:8220, where 192.0.2.1 is the host IP where you will install Fleet Server. Save and apply your settings.
In the Elasticsearch hosts field, specify the Elasticsearch URLs where Elastic Agents will send data. For example, https://192.0.2.0:9200. Skip this step if you’ve started the Elastic Stack with security enabled (you cannot change this setting because it’s managed outside of Fleet).
Save and apply the settings.
Click the Agents tab and follow the in-product instructions to add a Fleet server:

Notes:

Choose Quick Start if you want Fleet to generate a Fleet Server policy and enrollment token for you. The Fleet Server policy will include a Fleet Server integration plus a system integration for monitoring Elastic Agent. This option generates self-signed certificates and is not recommended for production use cases.
Choose Advanced if you want to either:
- Use your own Fleet Server policy. You can create a new Fleet Server policy or select an existing one. Alternatively you can create a Fleet Server policy without using the UI, and select the policy here.
- Use your own TLS certificates to encrypt traffic between Elastic Agents and Fleet Server. To learn how to generate certs, refer to Configure SSL/TLS for self-managed Fleet Servers.
It’s recommended you generate a unique service token for each Fleet Server. For other ways to generate service tokens, see elasticsearch-service-tokens.
If you are providing your own certificates:
- Before running the install command, make sure you replace the values in angle brackets.
- Note that the URL specified by --url must match the DNS name used to generate the certificate specified by --fleet-server-cert.
The install command installs the Elastic Agent as a managed service and enrolls it in a Fleet Server policy. For more Fleet Server commands, see Elastic Agent command reference.

If installation is successful, you’ll see confirmation that Fleet Server connected. Click Continue enrolling Elastic Agent to begin enrolling your agents in Fleet Server.

If you’re unable to add a Fleet-managed agent, click the Agents tab and confirm that the agent running Fleet Server is healthy.

For more information, refer to Fleet Server.

Step 2: Add the Elastic Agent System integration

edit

Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, forward data from remote services or hardware, and more. A single agent makes it easier and faster to deploy monitoring across your infrastructure. Each agent has a single policy you can update to add integrations for new data sources, security protections, and more.

In this step, add the System integration to monitor host logs and metrics.

Go to the Kibana home page and click Add integrations.
In the query bar, search for System and select the integration to see more details about it.
Click Add System.
Configure the integration name and optionally add a description. Make sure that Collect logs from System instances and Collect metrics from System instances are turned on.
Expand each configuration section to verify that the settings are correct for your host. For example, if you’re deploying Elastic Agent on macOS hosts, you need to add a new path to the System syslog logs section by clicking Add row and specifying /var/log/system.log.
Click Save and continue. This step takes a minute or two to complete. When it’s done, you’ll have an agent policy that contains a system integration policy for the configuration you just specified.
In the popup, click Add Elastic Agent to your hosts to open the Add agent flyout.

If you accidentally close the popup, go to Fleet > Agents, then click Add agent to access the flyout.

Step 3: Install and run an Elastic Agent on your machine

edit

The Add agent flyout has two options: Enroll in Fleet and Run standalone. The default is to enroll the agents in Fleet, as this reduces the amount of work on the person managing the hosts by providing a centralized management tool in Kibana.

Skip the Select enrollment token step. The enrollment token you need is already selected.

The enrollment token is specific to the Elastic Agent policy that you just created. When you run the command to enroll the agent in Fleet, you will pass in the enrollment token.
Download, install, and enroll the Elastic Agent on your host by selecting your host operating system and following the Install Elastic Agent on your host step.

It takes about a minute for Elastic Agent to enroll in Fleet, download the configuration specified in the policy you just created, and start collecting data.

Step 4: Monitor host logs and metrics

edit

Verify that data is flowing. Wait until agent enrollment is confirmed and incoming data is received, then click View assets to access dashboards related to the System integration.
Choose a dashboard that is related to the operating system of your monitored system. Dashboards are available for Microsoft Windows systems and Unix-like systems (for example, Linux and macOS).
Open the [Metrics System] Host overview dashboard to view performance metrics from your host system.

You can hover over any visualization to adjust its settings, or click the Edit button to make changes to the dashboard. To learn more, refer to Dashboard and visualizations.

Step 5: Monitor services using real browsers and lightweight HTTP, TCP, and ICMP checks

edit

[beta] This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. Next, you’ll add the Elastic Synthetics integration, enabling you to monitor the status and response times of applications and services in real time. You can monitor the availability of network endpoints via HTTP, TCP, ICMP or Browser monitors.

Add the Elastic Synthetics integration to your agent policy. You use policies to manage settings across a group of agents. An agent policy may contain any number of integrations for collecting observability data from the various services running on your host.

In Kibana, go to the Integrations page (click Add integrations in the home page or main menu).
In the query bar, search for Elastic Synthetics and select the integration to see more details about it.
Click Add Elastic Synthetics.

Configure the integration name and select your desired monitor type from the following monitor types:

HTTP	Connects via HTTP and verifies that the host returns the expected response. For detailed information about HTTP options, refer to our Heartbeat documentation.
TCP	Connects via TCP and verifies the endpoint by sending and receiving a custom payload. By default, the hostname and port are required. For detailed information about TCP options, refer to our Heartbeat documentation.
ICMP	Uses an ICMP `v4` and `v6` Echo Request to ping the configured hosts. By default, the host name is required. For detailed information about ICMP options, refer to our Heartbeat documentation.
Browser	Runs automated tests using a real Chromium browser via the synthetics agent. For detailed information about browser options, refer to our Heartbeat documentation. To create a browser monitor, you must use the elastic-agent-complete Docker container as this contains the dependencies necessary to run browser monitors. To learn more, refer to Set up monitors.

Enter the URL you want to monitor for availability, and select a monitor interval in seconds or minutes. By default, a monitoring schedule of every 3 minutes is selected.
The HTTP and TCP monitor types both support TLS. Under TLS settings, select Enable TLS configuration. Click the down arrow next to advanced HTTP or TCP options, and then enter your required settings.
Under Where to add this integration, select Existing hosts, then select the agent policy you created earlier. That way, you can deploy the change to the agent that’s already running.
When you’re done, click Save and continue, then Save and deploy changes.
To see the updated policy, click the agent policy link, for example, Agent policy 1.

The newly added Elastic Synthetics integration should appear on the Integrations tab in the agent policy, along with the System integration.

Any Elastic Agents assigned to this policy will collect logs, metrics, and uptime data from the host.
To view the data in the Uptime app, go to Observability > Uptime in the main menu.

Step 5: Monitor Nginx logs and metrics

edit

Next, add an Nginx integration to the policy used by your agent.

For these steps, we assume that you have nginx running on your host, and want to collect logs and metrics from it. If not, you can skip this part of the guide.

In Kibana, go to the Integrations page.
In the query bar, search for Nginx and select the integration to see more details about it.
Click Add Nginx.
Configure the integration name and optionally add a description.
Expand each configuration section to verify that the settings are correct for your host. You may need to change the Paths settings.
Under Where to add this integration, select Existing hosts, then select the agent policy you created earlier. That way, you can deploy the change to the agent that’s already running.
When you’re done, click Save and continue, then Save and deploy changes.
To see the updated policy, click the agent policy link.

The newly added Nginx integration should appear on the Integrations tab in your agent policy, along with the System and Elastic Synthetics integrations.

Any Elastic Agents assigned to this policy will collect logs and metrics from the Nginx server and the host, along with system logs and uptime data.
To view the data, go to Management > Fleet, then click the Data streams tab.
In the Actions column, navigate to the dashboards corresponding to the data stream.

What’s next?

edit

Now that data is streaming into the Elastic Stack, take your investigation to a deeper level! Use Elastic Observability to unify your logs, metrics, uptime, and application performance data.
Want to protect your endpoints from security threats? Try Elastic Security. Adding endpoint protection is just another integration that you add to the agent policy!
Are your eyes bleary from staring at a wall of screens? Create alerts and find out about problems while sipping your favorite beverage poolside.
Want Elastic to do the heavy lifting? Use machine learning to detect anomalies.
Got everything working like you want it? Roll out your agent policies to other hosts by deploying Elastic Agents across your infrastructure!

« Deploy Elastic Agent to send data Ingest application traces with Elastic Agent »