Securely manage credentials while monitoring Kubernetes workloads with autodiscovery

By
Christos Markou

In the world of containers and Kubernetes, observability is crucial. Cluster administrators need visibility into the infrastructure and cluster operators need to know the status of their workloads at any given time. And in both cases, they need observability into moving objects.

This is where Metricbeat and its autodiscover feature do the hard part for you. Operators can set up fairly complex monitoring scenarios with a self-service approach following the principle of least privilege at the cluster level.

The self-service approach is great, but what if the service I want to monitor is behind authentication? In this blog, we’ll take a look at how easy it is to configure observability for authenticated services with Elastic.

How autodiscover works

Applications running on containers and pods become moving targets to the monitoring system. Autodiscover tracks changes at the container and pod level and dynamically adjusts the monitoring configuration using two strategies:

  1. Template-based autodiscover: By defining configuration templates, the autodiscover subsystem can use them to monitor services as they start running.
  2. Hints-based autodiscover: The hints system looks for hints in Kubernetes pod annotations or Docker labels which have the prefix co.elastic.metrics. As soon as the container starts, Metricbeat will check if it contains any hints and launch the proper configuration for it. Hints tell Metricbeat how to get metrics for the given container.

Let’s look at an example with template-based autodiscover. Here is a configuration template for automatically detecting Redis pods and starting to collect metrics using the respective Redis Metricbeat module: 

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "redis"
          config:
            - module: redis
              metricsets: ["info"]
              hosts: "${data.host}:6379"

Defining autodiscover templates at Metricbeat’s configuration level is a valid approach, but users must be granted the permission to globally configure Metricbeat. In addition, a restart of Metricbeat is required every time the autodiscover template is updated. With hints-based autodiscover, we let workload owners enable monitoring for their services on the fly, avoiding over-granted permissions without restarting the Beats. The following example configures the same Redis monitoring from above using hints: 

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true

In this case, Redis pods need to be annotated accordingly: 

apiVersion: v1
kind: Pod
metadata:
  name: annotations-demo
  annotations:
    co.elastic.metrics/module: redis
    co.elastic.metrics/metricsets: info
    co.elastic.metrics/hosts: '${data.host}:6379'

Using passwords with autodiscover

In many cases, the target service requires password authentication in order to provide monitoring metrics. For instance, Redis can be configured to be password protected, meaning we have to provide a password to the Redis module in Metricbeat in order to collect metrics successfully. With template-based autodiscover, we would configure the template like this: 

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "redis"
          config:
            - module: redis
              metricsets: ["info"]
              hosts: "${data.host}:6379"
              password: 'myred1sp@ss'

Similarly, with hints-based autodiscover, we would annotate the pod with a string containing the password: 

apiVersion: v1
kind: Pod
metadata:
  name: annotations-demo
annotations:
  co.elastic.metrics/module: redis
  co.elastic.metrics/metricsets: info
  co.elastic.metrics/hosts: '${data.host}:6379'
  co.elastic.metrics/password: 'myred1sp@ss'

While we can use these setups for development or testing purposes, storing passwords in plain text poses a serious risk and should be avoided. We could put the password in an environment variable and let Metricbeat interpolate the manifest, but this wouldn’t make our security posture any better. Let’s see a preferable alternative.

The Metricbeat keystore

Metricbeat provides a feature called keystore that can be used to securely store sensitive information and reference it in the Metricbeat configuration as a secret. Let’s see how we can leverage this feature in our autodiscover template configurations. First, we create the keystore and add the password for Redis under the name REDIS_PASSWORD: 

metricbeat keystore create
metricbeat keystore add REDIS_PASSWORD

And provide “myred1sp@ss” when prompted.

Then the configuration template will be able to reference the string REDIS_PASSWORD to access the actual password: 

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "redis"
          config:
            - module: redis
              metricsets: ["info"]
              hosts: "${data.host}:6379"
              password: "${REDIS_PASSWORD}"

And that’s all! We managed to avoid using plain text passwords or defining ENV variables, and instead we leveraged the Metricbeat keystore to securely store the password for the target Redis pod.

Unfortunately, we can't use this strategy with hints-based autodiscovery — it wouldn’t be safe to expose the keystore to the hints mechanism because it could be a target for an attacker trying to exploit the whole keystore just using hints annotations. This is why we introduced a completely new Keystore, the Kubernetes secrets keystore.

The Kubernetes Secrets Keystore

With the Kubernetes Secrets Keystore we allow users to consume Kubernetes secrets directly in autodiscover configuration. It can be used for both template-based and hints-based autodiscover.

Users can refer a Kubernetes Secret with the following pattern: 

${kubernetes.<namespace>.<secret_name>.<value>}

For instance, ${kubernetes.default.redisPass.value} where kubernetes.default.redisPass.value specifies a key stored as a Kubernetes secret with the following:

  1. Kubernetes Namespace: default
  2. Secret Name: redisPass
  3. Secret Data Key: value

Note that pods can only consume secrets that belong to the same Kubernetes namespace. For example if our Redis pod is running in the staging namespace, it cannot access a secret created in the testing namespace.

Let's see how we could leverage the new Keystore backend. First we need to create the secret: 

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: redisPass
type: Opaque
data:
  value: $(echo -n "myred1sp@ss" | base64)
EOF

Now we can consume the redisPass secret from within our hint based autodiscover configuration: 

apiVersion: v1
kind: Pod
metadata:
  name: annotations-demo
  app: redis
annotations:
  co.elastic.metrics/module: redis
  co.elastic.metrics/metricsets: info
  co.elastic.metrics/hosts: '${data.host}:6379'
  co.elastic.metrics/password: '${kubernetes.default.redisPass.value}'

Kubernetes secrets keystore also works with autodiscover templates and we can access Kuberentes secrets from our static configurations, too: 

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "redis"
          config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:6379"
              password: "${kubernetes.default.redisPass.value}"

Conclusions

If you find the idea of securing sensitive configurations in your Kubernetes environment of utmost importance, then you will love Metricbeat keystore and Kubernetes secrets keystore — they allow you to securely connect without leaking plain text passwords.

Download Metricbeat and start storing your passwords securely for your autodiscovered workloads without worrying about leaking sensitive data. And make sure your data stays safe by using Elasticsearch Service on Elastic Cloud, where security comes standard. Start a free trial today.

If you have any questions, we are always happy to help on the Discuss forums.