Configuration
editConfiguration
editUpgrade the Beat specification
editYou can upgrade the Beat version or change settings by editing the YAML specification. ECK applies the changes by performing a rolling restart of the Beat Pods. Depending on the specification settings that you used, ECK will set the output part of the config, perform Kibana dashboard setup, restart Beats on certificates rollover and set up the Beats keystore.
Customize Beat configuration
editThe Beat configuration is defined in the config
element:
apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: quickstart spec: type: heartbeat version: 8.15.3 elasticsearchRef: name: quickstart config: heartbeat.monitors: - type: tcp schedule: '@every 5s' hosts: ["quickstart-es-http.default.svc:9200"] deployment: podTemplate: spec: dnsPolicy: ClusterFirstWithHostNet securityContext: runAsUser: 0
Alternatively, it can be provided through a Secret specified in the configRef
element:
apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: heartbeat-quickstart spec: type: heartbeat version: 8.15.3 elasticsearchRef: name: quickstart configRef: secretName: heartbeat-config deployment: podTemplate: spec: dnsPolicy: ClusterFirstWithHostNet securityContext: runAsUser: 0 --- apiVersion: v1 kind: Secret metadata: name: heartbeat-config stringData: beat.yml: |- heartbeat.monitors: - type: tcp schedule: '@every 5s' hosts: ["quickstart-es-http.default.svc:9200"]
For more details, check the Beats configuration section.
Customize the connection to an Elasticsearch cluster
editThe elasticsearchRef
element allows ECK to automatically configure Beats to establish a secured connection to a managed Elasticsearch cluster. By default it targets all nodes in your cluster. If you want to direct traffic to specific nodes of your Elasticsearch cluster, refer to Traffic Splitting for more information and examples.
Deploy a Beat
editECK supports the deployment of the following Beats:
For each Beat you want to deploy, you can specify the type
and version
elements. ECK creates a new user in Elasticsearch with a minimal set of appropriate roles and permissions to enable the use of all Beats features.
Deploy a Community Beat
editECK supports the deployment of any Community Beat.
-
Specify the
type
andversion
elements. -
Set the
image
element to point to the image to be deployed. -
Make sure the following roles exist in Elasticsearch:
-
If
elasticsearchRef
is provided, create the roleeck_beat_es_$type_role
, where$type
is the Beat type. For example, when deployingkafkabeat
, the role name iseck_beat_es_kafkabeat_role
. This role must have the permissions required by the Beat. Check the Elasticsearch documentation for more details. -
If
kibanaRef
is provided, create the roleeck_beat_kibana_$type_role
with the permissions required to setup Kibana dashboards.
-
If
Alternatively, create a user in Elasticsearch and include the credentials in the Beats config
for Elasticsearch output, Kibana setup or both. If elasticsearchRef
and kibanaRef
are also defined, ECK will use the provided user credentials when setting up the connections.
Set up Kibana dashboards
editECK can instruct Beats to set up example dashboards packaged with the Beat. To enable this, set the kibanaRef
element in the specification to point to ECK-managed Kibana deployment:
apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: quickstart spec: kibanaRef: name: quickstart ...
ECK will create a new user in Elasticsearch with a minimal set of appropriate roles and permissions that is needed for dashboard setup.
Secrets keystore for secure settings
editBeats offer a keystore for sensitive settings like passwords. This avoids storing them in plaintext in the configuration.
ECK exposes that mechanism with the secureSettings
element in the specification. Similar to Elasticsearch, you can use Kubernetes Secrets to provide the settings securely:
apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: quickstart spec: secureSettings: - secretName: agent-name-secret config: name: ${AGENT_NAME_VAR} ... --- apiVersion: v1 kind: Secret metadata: name: agent-name-secret stringData: AGENT_NAME_VAR: id_007
Check Beats documentation for more details.
Set Beat output
editIf the elasticsearchRef
element is specified, ECK populates the output section of the Beat config. ECK creates a user with appropriate roles and permissions and uses its credentials. If required, it also mounts the CA certificate in all Beat Pods, and recreates Pods when this certificate changes. Moreover, elasticsearchRef
element can refer to an ECK-managed Elasticsearch cluster by filling the name
, namespace
, serviceName
fields accordingly, as well as to a Kubernetes secret that contains the connection information to an Elasticsearch cluster not managed by it. In the latter case, for authenticating against the Elasticsearch cluster the secret must contain the fields of url
and either the username
with password
or the api-key
.
Output can be set to any value that is supported by a given Beat. To use it, remove the elasticsearchRef
element from the specification and include an appropriate output configuration in the config
or configRef
elements.
apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: quickstart spec: config: output.kafka: hosts: ["kafka1.default.svc:9092", "kafka2.default.svc:9092"] topic: '%{[fields.log_topic]}' partition.round_robin: reachable_only: false required_acks: 1 ...
Choose the deployment model
editDepending on the use case, Beats may need to be deployed as a Deployment or a DaemonSet. Provide a podTemplate
element under either the deployment
or the daemonSet
element in the specification to choose how a given Beat should be deployed. When choosing the deployment
option you can additionally specify the strategy used to replace old Pods with new ones.
apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: quickstart spec: deployment: strategy: type: Recreate podTemplate: spec: securityContext: runAsUser: 0
Consider picking the Recreate
strategy if you are using a hostPath
volume as the Beats data directory to avoid two Pods competing for the same directory.
Role Based Access Control for Beats
editSome Beats features (such as autodiscover or Kubernetes module metricsets) require that Beat Pods interact with Kubernetes APIs. Specific permissions are needed to allow this functionality. Standard Kubernetes RBAC rules apply. For example, to allow for autodiscover:
apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: quickstart spec: config: filebeat: autodiscover: providers: - node: ${NODE_NAME} type: kubernetes hints: enabled: true default_config: type: container paths: - /var/log/containers/*${data.kubernetes.container.id}.log daemonSet: podTemplate: spec: serviceAccount: elastic-beat-filebeat-quickstart automountServiceAccountToken: true containers: - name: filebeat env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName ... --- apiVersion: v1 kind: ServiceAccount metadata: name: elastic-beat-filebeat-quickstart --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: elastic-beat-autodiscover-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: elastic-beat-autodiscover subjects: - kind: ServiceAccount name: elastic-beat-filebeat-quickstart namespace: default --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: elastic-beat-autodiscover rules: - apiGroups: - "" resources: - nodes - namespaces - events - pods verbs: - get - list - watch - apiGroups: ["apps"] resources: - replicasets verbs: - get - list - watch - apiGroups: ["batch"] resources: - jobs verbs: - get - list - watch
Deploying Beats in secured clusters
editTo deploy Beats in clusters with the Pod Security Policy admission controller enabled, or in OpenShift clusters, you must grant additional permissions to the Service Account used by the Beat Pods. Those Service Accounts must be bound to a Role or ClusterRole that has use
permission for the required Pod Security Policy or Security Context Constraints. Different Beats and their features might require different settings set in their PSP/SCC.