Running Filebeat on Kubernetes

Filebeat Docker images can be used on Kubernetes to retrieve and ship container logs.

By deploying Filebeat as a DaemonSet we ensure we get a running instance on each node of the cluster.

Docker logs host folder (/var/lib/docker/containers) is mounted on the Filebeat container. Filebeat will start an input for these files and start harvesting them as they appear.

Everything is deployed under kube-system namespace, you can change that by updating the YAML file.

To get the manifests just run:

curl -L -O https://raw.githubusercontent.com/elastic/beats/6.3/deploy/kubernetes/filebeat-kubernetes.yaml

Some parameters are exposed in the manifest to configure logs destination, by default they will use an existing Elasticsearch deploy if it’s present, but you may want to change that behavior, so just edit the YAML file and modify them:

- name: ELASTICSEARCH_HOST
  value: elasticsearch
- name: ELASTICSEARCH_PORT
  value: "9200"
- name: ELASTICSEARCH_USERNAME
  value: elastic
- name: ELASTICSEARCH_PASSWORD
  value: changeme

To deploy Filebeat to Kubernetes just run:

kubectl create -f filebeat-kubernetes.yaml

Then you should be able to check the status by running:

$ kubectl --namespace=kube-system get ds/filebeat

NAME       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR   AGE
filebeat   32        32        0         32           0           <none>          1m

Logs should start flowing to Elasticsearch, all annotated with Add Kubernetes metadata processor.