Troubleshooting
editTroubleshooting
editBeat Pods are crashing when kibanaRef is specified
editWhen kibanaRef
is specified, Beat tries to connect to the Kibana instance. If it’s unable to do so, the Beat process exits and the Pod restarts. This may happen when Kibana is not yet up or when a Beat user is not yet created in Elasticsearch. The Pod may restart a few times when it is first deployed. Afterwards, the Beat should run successfully.
Configuration containing key: null is malformed
editWhen kubectl
is used to modify a resource, it calculates the diff between the user applied and the existing configuration. This diff has special semantics that forces the removal of keys if they have special values. For example, if the user-applied configuration contains some_key: null
(or equivalent some_key: ~
), this is interpreted as an instruction to remove some_key
. In Beats configurations, this is often a problem when it comes to defining things like processors. To avoid this problem:
-
Use
some_key: {}
(empty map) orsome_key: []
(empty array) instead ofsome_key: null
if doing so does not affect the behaviour. This might not be possible in all cases as some applications distinguish between null values and empty values and behave differently. -
Instead of using
config
to define configuration inline, useconfigRef
and store the configuration in a Secret.
Pod fails to start after update
editIf you have configured a Beat to run as a Deployment
and you are using a hostPath
volume as the Beats data directory, you might encounter an error similar to the following:
ERROR instance/beat.go:958 Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
This can happen if the new Pod is scheduled on the same Kubernetes node as the old Pod and is now trying to use the same data directory. Use a Recreate
deployment strategy to avoid this problem.