Tutorial: Observe your Kubernetes deployments

Applications running in a containerized environment like Kubernetes pose a unique monitoring challenge: how do you diagnose and resolve issues with hundreds of microservices on thousands (or millions) of containers, running in ephemeral and disposable pods?

A successful Kubernetes monitoring solution has a few requirements:

This guide describes how to use Elastic Observability to observe all layers of your application, including the orchestration software itself:

This guide describes how to deploy Elastic monitoring agents as DaemonSets using the Elastic Agent manifest files. For other deployment options, see the Kubernetes operator and custom resource definitions from Elastic Cloud on Kubernetes (ECK).

The Elastic Stack provides the following components for monitoring Kubernetes:

The default installation of Elastic Agent is deployed to Kubernetes as a DaemonSet to ensure an instance is running on each node of the cluster. It collects logs and metrics from pods, containers, and applications running on Kubernetes.

Elastic Agent provides processors for adding metadata to events. The metadata is valuable for grouping and exploring related data. For example, when analyzing container logs, you want to know the host and container name, and be able to correlate logs, metrics, and traces.

The default deployments include processors, when needed, for enriching events with cloud and host metadata.

For more on these processors, refer to the add_cloud_metadata and add_host_metadata documentation.

By default, the Kubernetes integration enriches logs and metrics with valuable metadata.

All Kubernetes metrics are enriched with metadata by default. The enrichment happens in code, and can be configured with the add_resource_metadata block on each dataset. For more information on configuring the add_resource_metadata block, refer to Configure kubelet API metadata enrichment and Configure kube-state-metrics metadata enrichment.

All Kubernetes logs are enriched with metadata by default. For more on configuring metadata enrichment, refer to Collect Kubernetes container logs.

Now that you have a basic understanding of the monitoring architecture, let’s learn how to deploy monitoring to your Kubernetes environment.

Before you can monitor Kubernetes, you need the following:

To start collecting logs and metrics from your Kubernetes clusters, first add the Kubernetes integration to your policy and configure the metrics and logs you want to collect.

Follow these steps to add the Kubernetes integration to your policy:

Continue to Step 2: Configure your Kubernetes integration.

The Kubernetes integration can fetch metrics and logs from several components. From the Add Kubernetes integration page, configure the options for the metrics and logs you want to collect. The following sections outline the configuration options.

Collect Metrics

Collecting metrics about Kubernetes clusters and the workloads running on top of them is a key aspect of Kubernetes observability. You need to collect metrics about the resources running on the underlying infrastructure (for example, the machines of your Kubernetes cluster), metrics for the whole cluster, as well as individual metrics for the containers and pods. Specifically, you need to monitor the health and performance of:

Elastic Agent along with the Kubernetes integration provides a unified solution to monitor all layers of your Kubernetes technology stack, so you don’t need multiple technologies to collect metrics.

You have the following options for collecting metrics about your Kubernetes clusters and the workloads running on top of them:

Collect logs

Collecting and analyzing logs of both Kubernetes core components and various applications running on top of Kubernetes is a powerful tool for Kubernetes observability. Containers running within Kubernetes pods publish logs to stdout or stderr.

You have the following options for collecting Kubernetes logs:

Collecting metrics from the kubelet API is on by default. Kubelet is an agent that runs on each Kubernetes node that is key to managing individual pods and the nodes that host them. For more information on kubelet, refer to the Kubernetes kubelet docs.

Elastic Agent along with the Kubernetes integration can collect metrics from the kubelet API to gather important information about the state of your Kubernetes nodes, pods, containers, and other resources. paleExpand the following list to see all available metrics from the kubelet API.

Provide the following information for each of the kubelet API options:

Under Kubernetes Container metrics and Kubernetes Pod metrics, you can configure metadata enrichment from Advanced options → Add node and namespace metadata.

From here, update the add_resource_metadata block to configure enrichment:

 add_resource_metadata:
   namespace:
     include_labels: ["namespacelabel1"]
   node:
     include_labels: ["nodelabel2"]
     include_annotations: ["nodeannotation1"]
   deployment: false

Collecting metrics from kube-state-metrics is on by default. The kube-state-metrics service provides cluster-wide metrics on the health of Kubernetes objects. Refer to the kube-state-metrics docs for more information.

kube-state-metrics provides horizontal sharding to support large Kubernetes deployments. Refer to the kube-state-metrics sharding documentation for more information.

With the Kubernetes integration, you can collect a number of metrics using the kube-state-metrics. Expand the following list to see all available metrics from kube-state-metrics.

Provide the following information for each of the kube-state-metrics options:

Under Kubernetes Container metrics and Kubernetes Pod metrics, you can configure metadata enrichment from Advanced options → Add node and namespace metadata.

From here, update the add_resource_metadata block to configure enrichment:

add_resource_metadata:
  namespace:
   enabled: true
    #use_regex_include: false
    include_labels: ["namespacelabel1"]
    #use_regex_exclude: false
    #exclude_labels: ["namespacelabel2"]
  node:
   enabled: true
    #use_regex_include: false
    include_labels: ["nodelabel2"]
    include_annotations: ["nodeannotation1"]
    #use_regex_exclude: false
    #exclude_labels: ["nodelabel3"]
  #deployment: false
  #cronjob: false

Collecting metrics from the kube-apiserver is on by default. The kube-apiserver sets up and validates pods, services, and other API objects. These metrics provide insight into the API server’s performance, workload, and health.

Refer to kube-apiserver metrics for more on the metrics collected.

Provide the following information to collect kube-apiserver metrics:

The kube-proxy runs on each node and maintains network rules. Collecting metrics from the kube-proxy is on by default. These metrics provide insight into the proxy’s networking activity, performance, and resource usage.

Refer to kube-proxy metrics for more on the metrics collected.

Provide the following information to collect Kubernetes Proxy metrics:

The kube-scheduler assigns new pods with no node assignment to the most appropriate node. Turn this option on to get metrics from the kube-scheduler. These metrics provide insight on the performance, resource usage, and health of the kube-scheduler.

Refer to kube-scheduler metrics for more on the metrics collected.

Provide the following information to collect Kubernetes scheduler metrics:

The kube-controller-manager regulates the state of the clusters. Turn this option on to get metrics from the kube-controller-manager. These metrics provide insight on the performance, resource usage, and health of the kube-controller-manager.

Refer to kube-controller-manager metrics for more on the metrics collected.

Provide the following information to collect kube-controller-manager metrics:

Event metrics give you an overall view of what’s happening in a cluster. These metrics help you understand what’s happening in your cluster and improve reliability and stability. Collecting Kubernetes events from the Kubernetes API server is on by default.

Refer to events metrics for more on the metrics collected.

Provide the following information to collect Kubernetes events metrics:

Collecting and parsing Kubernetes container logs is on by default. Containers running within Kubernetes pods publish logs to stdout or stderr. These logs are written to a location known to kubelet. The container parser is enabled by default. You can enable additional parsers in advanced settings.

Metadata enrichment is also enabled by default, and is based on the Kubernetes provider. Use the add_resource_metadata block of the Kubernetes provider to configure it. Refer to the Kubernetes provider docs for more on configuring the provider.

Refer to Kubernetes container logs for more on collecting container logs.

Provide the following information to collect container logs:

Turn this option on to collect audit logs. Kubernetes audit logs record requests that come to the Kubernetes API from internal and external components. These logs help you understand cluster behavior and debug issues.

Refer to Kubernetes audit logs for more on collecting audit logs.

After configuring your integration, you need to download and update your manifest. First, download the manifest by completing the following steps:

After downloading the manifest, open it and update the ES_USERNAME and ES_PASSWORD environment variables in the DaemonSet to match your Elasticsearch credentials.

You can also further modify the manifest to fit your needs. For example, you might want to enable autodiscovery to automatically discover container logs. Refer to the autodiscovery docs in the Fleet guide for more on enabling autodiscovery in your manifest.

Once you are ready to deploy your Elastic Agent:

Refer to Debug standalone Elastic Agents if you run into any issues with configuring or installing your Elastic Agent.

Use Kibana to view the metric and log data collected by Elastic Agent. Refer to the following sections for more on viewing your data.

To view the performance and health metrics collected by Elastic Agent, find Infrastructure in the main menu or use the global search field.

On the Infrastructure inventory page, you can switch between different views to see an overview of the containers and pods running on Kubernetes:

For more on using the Inventory page, refer to View infrastructure metrics by resource type.

On the Metrics Explorer page, you can group and analyze metrics for the resources that you are monitoring.

For more on using the Metrics Explorer page, refer to Explore infrastructure metrics over time.

Find Logs Explorer in the global search field.

With Logs Explorer, you can quickly search and filter your log data, get information about the structure of log fields, and display your findings in a visualization.

From Logs Explorer, you can select the Kubernetes integration from the data selector to view your Kubernetes data.

From here, you can filter your log data and dive deeper into individual logs to find and troubleshoot issues. For more information, refer to:

Quickly triage and troubleshoot application performance problems with the help of Elastic application performance monitoring (APM).

Think of a latency spike — APM can help you narrow the scope of your investigation to a single service. Because you’ve also ingested and correlated logs and metrics, you can then link the problem to CPU and memory utilization or error log entries of a particular Kubernetes pod.

Application monitoring data is streamed from your applications running in Kubernetes to APM, where it is validated, processed, and transformed into Elasticsearch documents.

There are many ways to deploy APM when working with Kubernetes, but this guide assumes that you’re using our hosted Elasticsearch Service on Elastic Cloud. If you haven’t done so already, enable APM in the Elasticsearch Service console.

If you want to manage APM yourself, there are a few alternative options:

A secret token is used to secure communication between APM agents and APM Server. To create or update your secret token in Kibana:

To avoid exposing the secret token, you can store it in a Kubernetes secret. For example:

kubectl create secret generic apm-secret --from-literal=ELASTIC_APM_SECRET_TOKEN=asecretpassword --namespace=kube-system 

If you’re managing APM Server yourself, see secret token for instructions on how to set up your secret token.

If you are using ECK to set up APM Server, the operator automatically generates an {APM-server-name}-apm-token secret for you.

In most cases, setting up APM agents and thereby instrumenting your applications is as easy as installing a library and adding a few lines of code.

Select your application’s language for details:

go get go.elastic.co/apm

import (
	"net/http"

	"go.elastic.co/apm/module/apmhttp"
)

func main() {
	mux := http.NewServeMux()
	...
	http.ListenAndServe(":8080", apmhttp.Wrap(mux))
}

        # ...
        - name: ELASTIC_APM_SERVER_URL
          value: "apm-server-url-goes-here" 
        - name: ELASTIC_APM_SECRET_TOKEN
          valueFrom:
            secretKeyRef:
              name: apm-secret
              key: ELASTIC_APM_SECRET_TOKEN 
        - name: ELASTIC_APM_SERVICE_NAME
          value: "service-name-goes-here" 

    # ...
    spec:
      volumes:
      - name: elastic-apm-agent 
        emptyDir: {}
      initContainers:
      - name: elastic-java-agent
        image: docker.elastic.co/observability/apm-agent-java:1.12.0 
        volumeMounts:
        - mountPath: /elastic/apm/agent
          name: elastic-apm-agent
        command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent'] 

      # ...
      containers:
      - name: your-app-container
        env:
        # ...
        - name: JAVA_TOOL_OPTIONS 
          value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar

        # ...
        - name: ELASTIC_APM_SERVER_URLS
          value: "apm-server-url-goes-here" 
        - name: ELASTIC_APM_SECRET_TOKEN
          valueFrom:
            secretKeyRef:
              name: apm-secret
              key: ELASTIC_APM_SECRET_TOKEN 
        - name: ELASTIC_APM_SERVICE_NAME
          value: "service-name-goes-here" 
        - name: ELASTIC_APM_APPLICATION_PACKAGES
          value: "org.springframework.samples.petclinic" 
        - name: JAVA_TOOL_OPTIONS 
          value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar

    # ...
    spec:
      volumes:
      - name: elastic-apm-agent 
        emptyDir: {}
      initContainers:
      - name: elastic-dotnet-agent
        image: busybox
        command: ["/bin/sh","-c"] 
        args: 
          - wget -qO './elastic-apm-agent/ElasticApmAgent.zip' https://github.com/elastic/apm-agent-dotnet/releases/download/1.7.0/ElasticApmAgent_1.7.0.zip;
            cd elastic-apm-agent;
            cat ElasticApmAgent.zip | busybox unzip -;
        volumeMounts:
        - mountPath: /elastic-apm-agent
          name: elastic-apm-agent

      # ...
      containers:
      - name: your-app-container
        volumeMounts:
        - mountPath: /elastic-apm-agent
          name: elastic-apm-agent
        env:
        # ...
        - name: DOTNET_STARTUP_HOOKS
          value: "/elastic-apm-agent/ElasticApmAgentStartupHook.dll"

        # ...
        - name: ELASTIC_APM_SERVER_URLS
          value: "apm-server-url-goes-here" 
        - name: ELASTIC_APM_SECRET_TOKEN
          valueFrom:
            secretKeyRef:
              name: apm-secret
              key: ELASTIC_APM_SECRET_TOKEN 
        - name: ELASTIC_APM_SERVICE_NAME
          value: "service-name-goes-here" 
        - name: DOTNET_STARTUP_HOOKS 
          value: "/elastic-apm-agent/ElasticApmAgentStartupHook.dll"

npm install elastic-apm-node --save

var apm = require('elastic-apm-node').start()

        # ...
        - name: ELASTIC_APM_SERVER_URL
          value: "apm-server-url-goes-here" 
        - name: ELASTIC_APM_SECRET_TOKEN
          valueFrom:
            secretKeyRef:
              name: apm-secret
              key: ELASTIC_APM_SECRET_TOKEN 
        - name: ELASTIC_APM_SERVICE_NAME
          value: "service-name-goes-here" 

rpm -ivh <package-file>.rpm

dpkg -i <package-file>.deb

apk add --allow-untrusted <package-file>.apk

        # ...
        - name: ELASTIC_APM_SERVER_URL
          value: "apm-server-url-goes-here" 
        - name: ELASTIC_APM_SECRET_TOKEN
          valueFrom:
            secretKeyRef:
              name: apm-secret
              key: ELASTIC_APM_SECRET_TOKEN 
        - name: ELASTIC_APM_SERVICE_NAME
          value: "service-name-goes-here" 

# Django
pip install elastic-apm

# Flask
pip install elastic-apm[flask]

INSTALLED_APPS = (
   # ...
   'elasticapm.contrib.django',
)

from elasticapm.contrib.flask import ElasticAPM

app = Flask(__name__)

apm = ElasticAPM(app)

        # ...
        - name: ELASTIC_APM_SERVER_URL
          value: "apm-server-url-goes-here" 
        - name: ELASTIC_APM_SECRET_TOKEN
          valueFrom:
            secretKeyRef:
              name: apm-secret
              key: ELASTIC_APM_SECRET_TOKEN 
        - name: ELASTIC_APM_SERVICE_NAME
          value: "service-name-goes-here" 

gem 'elastic-apm'

# config.ru

app = lambda do |env|
  [200, {'Content-Type' => 'text/plain'}, ['ok']]
end

# Wraps all requests in transactions and reports exceptions
use ElasticAPM::Middleware

# Start an instance of the Agent
ElasticAPM.start()

run app

# Gracefully stop the agent when process exits.
# Makes sure any pending transactions have already sent.
at_exit { ElasticAPM.stop }

        # ...
        - name: ELASTIC_APM_SERVER_URL
          value: "apm-server-url-goes-here" 
        - name: ELASTIC_APM_SECRET_TOKEN
          valueFrom:
            secretKeyRef:
              name: apm-secret
              key: ELASTIC_APM_SECRET_TOKEN 
        - name: ELASTIC_APM_SERVICE_NAME
          value: "service-name-goes-here" 

In most instances, APM agents automatically read Kubernetes data from inside the container and send it to APM Server. If this is not the case, or if you wish to override this data, you can set environment variables for the agents to read. These environment variable are set via the Downward API in your Kubernetes pod spec:

      # ...
      containers:
      - name: your-app-container
        env:
        # ...
        - name: KUBERNETES_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: KUBERNETES_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: KUBERNETES_POD_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid

The table below maps these environment variables to the APM metadata event field:

APM agents are deployed with your application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <<your-app>>
  namespace: kube-system
  labels:
    app: <<your-app>>
    service: <<your-app>>
spec:
  replicas: 1
  selector:
    matchLabels:
      app: <<your-app>>
  template:
    metadata:
      labels:
        app: <<your-app>>
        service: <<your-app>>
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      volumes:
      - name: elastic-apm-agent
        emptyDir: {}
      initContainers:
      - name: elastic-java-agent
        image: docker.elastic.co/observability/apm-agent-java:1.12.0
        volumeMounts:
        - mountPath: /elastic/apm/agent
          name: elastic-apm-agent
        command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent']
      containers:
      - name: <<your-app>>
        image: <<your-app>>
        volumeMounts:
        - mountPath: /elastic/apm/agent
          name: elastic-apm-agent
        env:
        - name: ELASTIC_APM_SERVER_URL
          value: "apm-server-url-goes-here"
        - name: ELASTIC_APM_SECRET_TOKEN
          valueFrom:
            secretKeyRef:
              name: apm-secret
              key: ELASTIC_APM_SECRET_TOKEN
        - name: ELASTIC_APM_SERVICE_NAME
          value: "petclinic"
        - name: ELASTIC_APM_APPLICATION_PACKAGES
          value: "org.springframework.samples.petclinic"
        - name: ELASTIC_APM_ENVIRONMENT
          value: test
        - name: JAVA_TOOL_OPTIONS
          value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar
        - name: KUBERNETES_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: KUBERNETES_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: KUBERNETES_POD_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid

kubectl apply -f demo.yml

Application trace data is available in the Service Inventory. To open Service Inventory, find Applications in the main menu or use the global search field.

The Applications app allows you to monitor your software services and applications in real-time: visualize detailed performance information on your services, identify and analyze errors, and monitor host-level and agent-specific metrics like JVM and Go runtime metrics.

Having access to application-level insights with just a few clicks can drastically decrease the time you spend debugging errors, slow response times, and crashes.

Best of all, because Kubernetes environment variables have been mapped to APM metadata events, you can filter your trace data by Kubernetes namespace, node.name, pod.name, and pod.uid.