Run Elastic Agent on Kubernetes managed by Fleet

Use Elastic Agent Docker images on Kubernetes to retrieve cluster metrics.

With Fleet, each agent enrolls in a policy defined in Kibana and stored in Elasticsearch. The policy specifies how to collect observability data from the services to be monitored. The Elastic Agent connects to a trusted Fleet Server instance to retrieve the policy and report agent events.

We recommend using Fleet management because it makes the management and upgrade of agents considerably easier.

On Kubernetes, deploy Elastic Agent as a DaemonSet to ensure that there is a running instance on each node of the cluster. These instances are used to retrieve metrics from the host, such as system metrics, container stats, and metrics from all the services running on top of Kubernetes.

In addition, one of the Pods in the DaemonSet will constantly hold a leader lock which makes it responsible for handling cluster-wide monitoring. Find more information about leader election configuration options at leader election provider. This instance is used to retrieve metrics that are unique for the whole cluster, such as Kubernetes events or kube-state-metrics.

Everything is deployed under the kube-system namespace by default. To change the namespace, modify the manifest file.

To download the manifest file, run:

curl -L -O https://raw.githubusercontent.com/elastic/beats/8.1/deploy/kubernetes/elastic-agent-managed-kubernetes.yaml

Elastic Agent is enrolled to a running Fleet Server using FLEET_URL parameter. The FLEET_ENROLLMENT_TOKEN parameter is used to connect Elastic Agent to a specific Elastic Agent policy. To learn how to get an enrollment token from Fleet, see Fleet enrollment tokens.

To specify different destination/credentials, change the following parameters in the manifest file:

- name: FLEET_URL
  value: "https://fleet-server_url:port"
- name: FLEET_ENROLLMENT_TOKEN
  value: "token"
- name: FLEET_SERVER_POLICY_ID
  value: "fleet-server-policy" 
- name: KIBANA_HOST
  value: ""
- name: KIBANA_FLEET_USERNAME
  value: ""
- name: KIBANA_FLEET_PASSWORD
  value: ""

Kubernetes master nodes can use taints to limit the workloads that can run on them. The manifest for managed Elastic Agent defines tolerations to run on master nodes. Agents running on master nodes collect metrics from the control plane components (scheduler, controller manager) of Kubernetes. To disable Elastic Agent from running on master nodes, remove the following part of the Daemonset spec:

spec:
 tolerations:
 - key: node-role.kubernetes.io/master
   effect: NoSchedule

If planning to deploy state_* datasets of Kubernetes package, kube-state-metrics needs to be already deployed in the cluster. If kube-state-metrics is not already running, deploy it now (see the Kubernetes deployment docs).

To deploy Elastic Agent on Kubernetes, run:

kubectl create -f elastic-agent-managed-kubernetes.yaml

To check the status, run:

$ kubectl get pod -n kube-system -l app=elastic-agent

NAME                  READY   STATUS    RESTARTS   AGE
elastic-agent-hrjbg   1/1     Running   0          12m
elastic-agent-olpsd   1/1     Running   0          12m

Elastic Agents should be enrolled to Fleet and users should be able to deploy the Kubernetes package accordingly. This can be confirmed in Kibana under Fleet / Agents section.

The size and the number of nodes in a Kubernetes cluster can be fairly large at times, and in such cases the Pod that will be collecting cluster level metrics might face performance issues due to resources limitations. In this case users might consider to avoid using the leader election strategy and instead run a dedicated, standalone Elastic Agent instance using a Deployment in addition to the DaemonSet.

On managed Kubernetes solutions, such as AKS, GKE or EKS, Elastic Agent has no access to collect metrics from the Kubernetes control plane components, like kube-scheduler and kube-controller-manager, that are scheduled on Kubernetes master nodes.

Audit logs are available only on Kubernetes master nodes as well and hence cannot be collected by Elastic Agent.