IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Run Filebeat on Docker Run Filebeat on Cloud Foundry »

› ›

Run Filebeat on Kubernetes

edit

Run Filebeat on Kubernetes

edit

You can use Filebeat Docker images on Kubernetes to retrieve and ship container logs.

Running Elastic Cloud on Kubernetes? See Run Beats on ECK.

Kubernetes deploy manifests

edit

You deploy Filebeat as a DaemonSet to ensure there’s a running instance on each node of the cluster.

The Docker logs host folder (/var/lib/docker/containers) is mounted on the Filebeat container. Filebeat starts an input for the files and begins harvesting them as soon as they appear in the folder.

Everything is deployed under the kube-system namespace by default. To change the namespace, modify the manifest file.

To download the manifest file, run:

curl -L -O https://raw.githubusercontent.com/elastic/beats/7.10/deploy/kubernetes/filebeat-kubernetes.yaml

If you are using Kubernetes 1.7 or earlier: Filebeat uses a hostPath volume to persist internal data. It’s located under /var/lib/filebeat-data. The manifest uses folder autocreation (DirectoryOrCreate), which was introduced in Kubernetes 1.8. You need to remove type: DirectoryOrCreate from the manifest and create the host folder yourself.

Settings

edit

By default, Filebeat sends events to an existing Elasticsearch deployment, if present. To specify a different destination, change the following parameters in the manifest file:

- name: ELASTICSEARCH_HOST
  value: elasticsearch
- name: ELASTICSEARCH_PORT
  value: "9200"
- name: ELASTICSEARCH_USERNAME
  value: elastic
- name: ELASTICSEARCH_PASSWORD
  value: changeme

Red Hat OpenShift configuration

edit

If you are using Red Hat OpenShift, you need to specify additional settings in the manifest file and enable the container to run as privileged.

Modify the DaemonSet container spec in the manifest file:

  securityContext:
    runAsUser: 0
    privileged: true

Grant the filebeat service account access to the privileged SCC:
```
oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:filebeat
```
This command enables the container to be privileged as an administrator for OpenShift.
Override the default node selector for the kube-system namespace (or your custom namespace) to allow for scheduling on any node:
```
oc patch namespace kube-system -p \
'{"metadata": {"annotations": {"openshift.io/node-selector": ""}}}'
```
This command sets the node selector for the project to an empty string. If you don’t run this command, the default node selector will skip master nodes.

Load Kibana dashboards

edit

Filebeat comes packaged with various pre-built Kibana dashboards that you can use to visualize logs from your Kubernetes environment.

If these dashboards are not already loaded into Kibana, you must install Filebeat on any system that can connect to the Elastic Stack, and then run the setup command to load the dashboards. To learn how, see Load Kibana dashboards.

The setup command does not load the ingest pipelines used to parse log lines. By default, ingest pipelines are set up automatically the first time you run Filebeat and connect to Elasticsearch.

If you are using a different output other than Elasticsearch, such as Logstash, you need to:

Deploy

edit

To deploy Filebeat to Kubernetes, run:

kubectl create -f filebeat-kubernetes.yaml

To check the status, run:

$ kubectl --namespace=kube-system get ds/filebeat

NAME       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR   AGE
filebeat   32        32        0         32           0           <none>          1m

Log events should start flowing to Elasticsearch. The events are annotated with metadata added by the add_kubernetes_metadata processor.

« Run Filebeat on Docker Run Filebeat on Cloud Foundry »