Configuration

Any setting can be changed in the Beat YAML specification, including version upgrades. ECK detects those changes and ensures a smooth rolling upgrade of all Beat Pods. Depending on specification settings used, ECK will set the output part of the config, perform Kibana dashboard setup, restart Beats on certificates rollover and set up the Beats keystore.

The Beat configuration is provided through the config element:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  type: heartbeat
  version: 8.16.1
  elasticsearchRef:
    name: quickstart
  config:
    heartbeat.monitors:
    - type: tcp
      schedule: '@every 5s'
      hosts: ["quickstart-es-http.default.svc:9200"]
  deployment:
    podTemplate:
      spec:
        dnsPolicy: ClusterFirstWithHostNet
        securityContext:
          runAsUser: 0

Alternatively, it can be provided via a Secret specified in the configRef element:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: heartbeat-quickstart
spec:
  type: heartbeat
  version: 8.16.1
  elasticsearchRef:
    name: quickstart
  configRef:
    secretName: heartbeat-config
  deployment:
    podTemplate:
      spec:
        dnsPolicy: ClusterFirstWithHostNet
        securityContext:
          runAsUser: 0
---
apiVersion: v1
kind: Secret
metadata:
  name: heartbeat-config
stringData:
  beat.yml: |-
    heartbeat.monitors:
    - type: tcp
      schedule: '@every 5s'
      hosts: ["quickstart-es-http.default.svc:9200"]

For more details about Beats configuration, see the Beats documentation.

ECK supports the deployment of the following Beats:

You can specify the Beat to deploy and its version through type and version elements. ECK creates a new user in Elasticsearch with a minimal set of appropriate roles and permissions to enable the use of all Beats features.

ECK supports the deployment of any Community Beat. type and version specification elements have to be provided. In addition:

Alternatively, create a user in Elasticsearch and include the credentials in the Beats config for Elasticsearch output, Kibana setup or both. If elasticsearchRef and kibanaRef are also defined, ECK will use the provided user credentials when setting up the connections.

ECK can instruct Beats to set up example dashboards packaged with the Beat. To enable this, set the kibanaRef element in the specification to point to ECK-managed Kibana deployment:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  kibanaRef:
    name: quickstart
...

ECK will create a new user in Elasticsearch with a minimal set of appropriate roles and permissions that is needed for dashboard setup.

Beats offer a secret keystore for sensitive settings that need to be provided in the config, for example passwords. This avoids storing them in the config directly.

ECK exposes that mechanism with the secureSettings element in the specification. Similar to Elasticsearch, you can use Kubernetes Secrets to provide the settings securely:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  secureSettings:
  - secretName: agent-name-secret
  config:
    name: ${AGENT_NAME_VAR}
...
---
apiVersion: v1
kind: Secret
metadata:
  name: agent-name-secret
stringData:
  AGENT_NAME_VAR: id_007

See Beats documentation for more details.

If the elasticsearchRef element is specified, ECK populates the output section of the Beat config. ECK creates a user with appropriate roles and permissions and uses its credentials. If required, it also mounts the CA certificate in all Beat Pods, and recreates Pods when this certificate changes.

Output can be set to any value that is supported by a given Beat. To use it, remove the elasticsearchRef element from the specification and include an appropriate output configuration in the config or configRef elements.

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  config:
    output.kafka:
      hosts: ["kafka1.default.svc:9092", "kafka2.default.svc:9092"]
      topic: '%{[fields.log_topic]}'
      partition.round_robin:
        reachable_only: false
      required_acks: 1
...

Depending on the use case, Beats may need to be deployed as a Deployment or a DaemonSet. Provide a podTemplate element under either the deployment or the daemonSet element in the specification to choose how a given Beat should be deployed.

Some Beats features (such as autodiscover or Kubernetes module metricsets) require that Beat Pods interact with Kubernetes APIs. Specific permissions are needed to allow this functionality. Standard Kubernetes RBAC rules apply. For example, to allow for autodiscover:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  config:
    filebeat:
      autodiscover:
        providers:
        - host: ${HOSTNAME}
          type: kubernetes
          hints:
            enabled: true
            default_config:
              type: container
              paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
  daemonSet:
    podTemplate:
      spec:
        serviceAccount: elastic-beat-filebeat-quickstart
        automountServiceAccountToken: true
...
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-beat-filebeat-quickstart
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-beat-autodiscover-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elastic-beat-autodiscover
subjects:
- kind: ServiceAccount
  name: elastic-beat-filebeat-quickstart
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-beat-autodiscover
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs:
  - get
  - list
  - watch

To deploy Beats in clusters with the Pod Security Policy admission controller enabled, or in OpenShift clusters, you must grant additional permissions to the Service Account used by the Beat Pods. Those Service Accounts must be bound to a Role or ClusterRole that has use permission for the required Pod Security Policy or Security Context Constraints. Different Beats and their features might require different settings set in their PSP/SCC.