Quick start: Get logs, metrics, and uptime data into the Elastic Stack

edit

Quick start: Get logs, metrics, and uptime data into the Elastic Stack

edit

This guide describes how to:

  • Set up Fleet
  • Send data to the Elastic Stack using Elastic Agent
  • Monitor logs and metrics from systems and services across your organization
  • Monitor the availability of your HTTP, TCP, and ICMP services using the Synthetics integration
  • Monitor Nginx logs and metrics using the Nginx integration

For feedback and questions, please contact us in the discuss forum.

Prerequisites

edit
  • You need Elasticsearch for storing and searching your data, and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud (recommended), or self-manage the Elastic Stack on your own hardware.

    Here’s what you need for each deployment type:

  • An internet connection is required for Kibana to download integration packages from the Elastic Package Registry. Make sure the Kibana server can connect to https://epr.elastic.co on port 443.
  • Fleet is currently only available to users with the superuser role.

Step 1: Set up Fleet

edit

Use Fleet in Kibana to get logs, metrics, and security data into the Elastic Stack.

Not using Fleet? Advanced users who want to configure and manage Elastic Agents manually can run agents standalone.

The first time you use Fleet, you might need to set it up and add a Fleet Server:

Elastic Cloud runs a hosted version of Fleet Server. No extra setup is required unless you want to scale your deployment.

To confirm that Fleet Server is available in your deployment:

  1. Log in to Kibana and go to Management > Fleet.
  2. Click the Agents tab.
  3. Under Agent policy, look for Elastic Cloud agent policy — Fleet Server is the agent enrolled in the Elastic Cloud agent policy. This policy is managed by Elastic Cloud. You cannot modify it. Confirm that the agent status is Healthy.

Don’t see the Fleet Server agent? Make sure your deployment includes an APM & Fleet node. This node is required to use Fleet Server.

Fleet Server hosted agent

For more information, see Fleet Server.

Step 2: Add an Elastic Agent to Fleet

edit

Elastic Agent is a single, unified agent that you can deploy to hosts or containers to collect data and send it to the Elastic Stack. Behind the scenes, Elastic Agent runs the Beats shippers or Elastic Endpoint required for your configuration.

To send logs and metrics to the Elastic Stack:

  1. On the Agents tab in Fleet, click Add agent.
  2. Under Enroll in Fleet, follow the in-product installation steps.

    add agent

    See the download page for other installation options.

Notes:

  • Use the default agent policy to get started quickly. This policy includes a system integration for collecting logs and metrics from the host system. You can change the policy later.
  • The install command installs the Elastic Agent as a managed service, enrolls it in the selected policy, and starts the service. For example:

    ./elastic-agent install -f --url=https://10.0.2.2:8220 \
    --enrollment-token=blJqaUdua0JqYXA0bmNscVVjUkE6ZGh4WWNRSHRRek9aSS1paEs2cHdFQQ==

    If you see an "x509: certificate signed by unknown authority" error, you might be trying to enroll in a Fleet Server that uses self-signed certs. To fix this problem in a non-production environment, pass the --insecure flag. For more information, refer to the troubleshooting guide.

  • Because Elastic Agent is installed as an auto-starting service, it will restart automatically if the system is rebooted.
  • To see where files are installed, see Installation layout.

If installation is successful, you’ll see the agent on the Agents tab in Fleet. Notice that the Default policy is assigned to the agent.

Fleet showing enrolled agents

If the status hangs at Enrolling, make sure the elastic-agent process is running.

If you run into problems:

Step 3: Monitor host logs and metrics

edit

Next, view the data sent by Elastic Agent. Right now, Elastic Agent is only sending data about the host system because you haven’t configured the agent to collect data from other sources yet.

To see host logs and metrics:

  1. In Fleet, click the Data streams tab.
  2. In the Actions column, navigate to the dashboards corresponding to the data stream. For example, to see host metrics, select one of the system datasets:

    Fleet showing data streams list

    Then navigate to the [Metrics System] Host overview dashboard:

    Host overview dashboard in Kibana

Step 4: Monitor HTTP, TCP, and ICMP services

edit

[beta] This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. Next, you’ll add the Elastic Synthetics integration, enabling you to monitor the status and response times of applications and services in real time. You can monitor the availability of network endpoints via HTTP, TCP, or ICMP.

Add the Elastic Synthetics integration to the default policy used by your agent. You use policies to manage settings across a group of agents. An agent policy may contain any number of integrations for collecting observability data from the various services running on your host.

  1. In Kibana, go to Management > Integrations, and search for the Elastic Synthetics integration.

    Fleet showing Synthetics integration
  2. Click the Elastic Synthetics integration to see more details about it, then click Add Elastic Synthetics.

    Fleet showing Synthetics integration overview
  3. On the Add Elastic Synthetics integration page, under Configure integration, enter the integration name and select HTTP from the following monitor types:

    HTTP

    Connects via HTTP and verifies that the host returns the expected response.

    For detailed information about HTTP options, see our Heartbeat documentation.

    TCP

    Connects via TCP and verifies the endpoint by sending and receiving a custom payload. By default, the hostname and port are required.

    For detailed information about TCP options, see our Heartbeat documentation.

    ICMP

    Uses an ICMP v4 and v6 Echo Request to ping the configured hosts. By default, the host name is required.

    For detailed information about ICMP options, see our Heartbeat documentation.

  4. Enter the URL you want to monitor for availability and select a monitor interval in seconds or minutes. By default, a monitoring schedule of every 3 minutes is selected.
  5. The HTTP and TCP monitor types both support TLS. Under TLS settings, select Enable TLS configuration. Click the down arrow next to advanced HTTP or TCP options, and then enter your required settings.
  6. Under Apply to agent policy, select the default policy.

    Fleet Add Synthetics integration page
  7. When you’re done, click Save integration, then Save and deploy changes.
  8. To see the updated policy, click the Default policy link.

    The newly added Elastic Synthetics integration should appear under Integrations in the default policy, along with the system-1 integration.

    Fleet showing default agent policy with synthetics-1 datasource

    All Elastic Agents that use this policy will collect logs, metrics, and uptime data from the host.

  9. To view the data in the Uptime app, go to Observability > Uptime.

Step 5: Monitor Nginx logs and metrics

edit

Next, you’ll browse a catalog of integrations, then add an Nginx integration to the default policy used by your agent.

For these steps, we assume that you have nginx running on some of your infrastructure, and want to collect logs and metrics from it.

  1. In Kibana, go back to Management > Integrations, and search for the Nginx integration.

    Fleet showing Nginx integration
  2. Click the Nginx integration to see more details about it, then click Add Nginx.

    Fleet showing Nginx integration overview
  3. On the Add Nginx integration page, select the default policy.

    Fleet Add Nginx integration page
  4. Under Configure integration, click the down arrow next to enabled streams and make sure the Paths are correct for your host. Inspect or change other settings
  5. When you’re done, save and deploy the changes.
  6. To see the updated policy, click the Default policy link.

    The newly added Nginx integration should appear under Integrations in the default policy, along with the system-1 and synthetics-1 integrations.

    Fleet showing default agent policy with nginx-1 datasource

    All Elastic Agents that use this policy will collect logs and metrics from the Nginx server and the host, along with uptime data.

  7. To view the data, click View all agent policies to return to the Fleet home page, then click the Data streams tab.
  8. In the Actions column, navigate to the dashboards corresponding to the data stream.

What’s next?

edit
  • Now that data is streaming into the Elastic Stack, take your investigation to a deeper level! Use Elastic Observability to unify your logs, metrics, uptime, and application performance data.
  • Want to protect your endpoints from security threats? Try Elastic Security. Adding endpoint protection is just another integration that you add to the agent policy!
  • Are your eyes bleary from staring at a wall of screens? Create alerts and find out about problems while sipping your favorite beverage poolside.
  • Want Elastic to do the heavy lifting? Use machine learning to detect anomalies.
  • Got everything working like you want it? Roll out your agent policies to other hosts by deploying Elastic Agents across your infrastructure!