« Run Elastic Agent in a container Install Elastic Agent on Kubernetes using Helm »

› › ›

Run Elastic Agent on Kubernetes managed by Fleet

edit

Run Elastic Agent on Kubernetes managed by Fleet

edit

What you need

edit

kubectl installed.
Elasticsearch for storing and searching your data, and Kibana for visualizing and managing it.

To get started quickly, spin up a deployment of our hosted Elasticsearch Service. The Elasticsearch Service is available on AWS, GCP, and Azure. Try it out for free.
kube-state-metrics.

You need to deploy kube-state-metrics to get the metrics about the state of the objects on the cluster (see the Kubernetes deployment docs). You can do that by first downloading the project:
```
gh repo clone kubernetes/kube-state-metrics
```
And then deploying it:
```
kubectl apply -k kube-state-metrics
```
On managed Kubernetes solutions, such as AKS, GKE or EKS, Elastic Agent does not have the required permissions to collect metrics from Kubernetes control plane components, like kube-scheduler and kube-controller-manager. Audit logs are only available on Kubernetes control plane nodes as well, and hence cannot be collected by Elastic Agent. Refer here to find more information. For more information about specific cloud providers, refer to Run Elastic Agent on Azure AKS managed by Fleet, Run Elastic Agent on GKE managed by Fleet and Run Elastic Agent on Amazon EKS managed by Fleet

Step 1: Download the Elastic Agent manifest

edit

You can find Elastic Agent Docker images here.

Download the manifest file:

curl -L -O https://raw.githubusercontent.com/elastic/elastic-agent/8.17/deploy/kubernetes/elastic-agent-managed-kubernetes.yaml

You might need to adjust resource limits of the Elastic Agent container in the manifest. Container resource usage depends on the number of data streams and the environment size.

This manifest includes the Kubernetes integration to collect Kubernetes metrics and System integration to collect system level metrics and logs from nodes.

The Elastic Agent is deployed as a DaemonSet to ensure that there is a running instance on each node of the cluster. These instances are used to retrieve most metrics from the host, such as system metrics, Docker stats, and metrics from all the services running on top of Kubernetes. These metrics are accessed through the deployed kube-state-metrics. Notice that everything is deployed under the kube-system namespace by default. To change the namespace, modify the manifest file.

Moreover, one of the Pods in the DaemonSet will constantly hold a leader lock which makes it responsible for handling cluster-wide monitoring. You can find more information about leader election configuration options at leader election provider. The leader pod will retrieve metrics that are unique for the whole cluster, such as Kubernetes events or kube-state-metrics.

For Kubernetes Security Posture Management (KSPM) purposes, the Elastic Agent requires read access to various types of Kubernetes resources, node processes, and files. To achieve this, read permissions are granted to the Elastic Agent for the necessary resources, and volumes from the hosting node’s file system are mounted to allow accessibility to the Elastic Agent pods.

The size and the number of nodes in a Kubernetes cluster can be large at times, and in such a case the Pod that will be collecting cluster level metrics might require more runtime resources than you would like to dedicate to all of the pods in the DaemonSet. The leader which is collecting the cluster wide metrics may face performance issues due to resource limitations if under-resourced. In this case users might consider avoiding the use of a single DaemonSet with the leader election strategy and instead run a dedicated standalone Elastic Agent instance for collecting cluster wide metrics using a Deployment in addition to the DaemonSet to collect metrics for each node. Then both the Deployment and the DaemonSet can be resourced independently and appropriately. For more information check the Scaling Elastic Agent on Kubernetes page.

Step 2: Configure Elastic Agent policy

edit

The Elastic Agent needs to be assigned to a policy to enable the proper inputs. To achieve Kubernetes observability, the policy needs to include the Kubernetes integration. Refer to Create a policy and Add an integration to a policy to learn how to configure the Kubernetes integration.

Step 3: Enroll Elastic Agent to the policy

edit

Enrollment of an Elastic Agent is defined as the action to register a specific agent to a running Fleet Server.

Elastic Agent is enrolled to a running Fleet Server by using FLEET_URL parameter. Additionally, the FLEET_ENROLLMENT_TOKEN parameter is used to connect Elastic Agent to a specific Elastic Agent policy.

A new FLEET_ENROLLMENT_TOKEN will be created upon new policy creation and will be inserted inside the Elastic Agent Manifest during the Guided installation.

Find more information for Enrollment Tokens.

To specify different destination/credentials, change the following parameters in the manifest file:

- name: FLEET_URL
  value: "https://fleet-server_url:port" 
- name: FLEET_ENROLLMENT_TOKEN
  value: "token" 
- name: FLEET_SERVER_POLICY_ID
  value: "fleet-server-policy" 
- name: KIBANA_HOST
  value: "" 
- name: KIBANA_FLEET_USERNAME
  value: "" 
- name: KIBANA_FLEET_PASSWORD
  value: ""

	URL to enroll the Fleet Server into. You can find it in Kibana. Select Management → Fleet → Fleet Settings, and copy the Fleet Server host URL.
	The token to use for enrollment. Close the flyout panel and select Enrollment tokens. Find the Agent policy you created before to enroll Elastic Agent into, and display and copy the secret token.
	The policy ID for Fleet Server to use on itself.
	The Kibana host.
	The basic authentication username used to connect to Kibana and retrieve a `service_token` to enable Fleet.
	The basic authentication password used to connect to Kibana and retrieve a `service_token` to enable Fleet.

If you need to run Fleet Server as well, adjust the docker run command above by adding these environment variables:

- name: FLEET_SERVER_ENABLE
  value: "true" 
- name: FLEET_SERVER_ELASTICSEARCH_HOST
  value: "<elasticsearch-host>" 
- name: FLEET_SERVER_SERVICE_TOKEN
  value: "<service-token>"

	Set to `true` to bootstrap Fleet Server on this Elastic Agent. This automatically forces Fleet enrollment as well.
	The Elasticsearch host for Fleet Server to communicate with, for example `http://elasticsearch:9200`.
	Service token to use for communication with Elasticsearch and Kibana.

Refer to Environment variables for all available options.

Step 4: Configure tolerations

edit

Kubernetes control plane nodes can use taints to limit the workloads that can run on them. The manifest for standalone Elastic Agent defines tolerations to run on these. Agents running on control plane nodes collect metrics from the control plane components (scheduler, controller manager) of Kubernetes. To disable Elastic Agent from running on control plane nodes, remove the following part of the DaemonSet spec:

spec:
  # Tolerations are needed to run Elastic Agent on Kubernetes control-plane nodes.
  # Agents running on control-plane nodes collect metrics from the control plane components (scheduler, controller manager) of Kubernetes
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      effect: NoSchedule

Both these two tolerations do the same, but node-role.kubernetes.io/master is deprecated as of Kubernetes version v1.25.

Step 5: Deploy the Elastic Agent

edit

To deploy Elastic Agent to Kubernetes, run:

kubectl create -f elastic-agent-managed-kubernetes.yaml

To check the status, run:

$ kubectl -n kube-system get pods -l app=elastic-agent
NAME                            READY   STATUS    RESTARTS   AGE
elastic-agent-4665d             1/1     Running   0          81m
elastic-agent-9f466c4b5-l8cm8   1/1     Running   0          81m
elastic-agent-fj2z9             1/1     Running   0          81m
elastic-agent-hs4pb             1/1     Running   0          81m

Running Elastic Agent on a read-only file system

If you’d like to run Elastic Agent on Kubernetes on a read-only file system, you can do so by specifying the readOnlyRootFilesystem option.

Step 6: View your data in Kibana

edit

Launch Kibana:
1. Log in to your Elastic Cloud account.
2. Navigate to the Kibana endpoint in your deployment.
Point your browser to http://localhost:5601, replacing localhost with the name of the Kibana host.
To check if your Elastic Agent is enrolled in Fleet, go to Management → Fleet → Agents.
To view data flowing in, go to Analytics → Discover and select the index metrics-*, or even more specific, metrics-kubernetes.*. If you can’t see these indexes, create a data view for them.
To view predefined dashboards, either select Analytics→Dashboard or install assets through an integration.

« Run Elastic Agent in a container Install Elastic Agent on Kubernetes using Helm »