Troubleshooting

Beat Pods are crashing when kibanaRef is specified

When kibanaRef is specified, Beat tries to connect to the Kibana instance. If it’s unable to do so, the Beat process exits and the Pod restarts. This may happen when Kibana is not yet up or when a Beat user is not yet created in Elasticsearch. The Pod may restart a few times when it is first deployed. Afterwards, the Beat should run successfully.

Configuration containing key: null is malformed

edit

When kubectl is used to modify a resource, it calculates the diff between the user applied and the existing configuration. This diff has special semantics that forces the removal of keys if they have special values. For example, if the user-applied configuration contains some_key: null (or equivalent some_key: ~), this is interpreted as an instruction to remove some_key. In Beats configurations, this is often a problem when it comes to defining things like processors. To avoid this problem:

Use some_key: {} (empty map) or some_key: [] (empty array) instead of some_key: null if doing so does not affect the behaviour. This might not be possible in all cases as some applications distinguish between null values and empty values and behave differently.
Instead of using config to define configuration inline, use configRef and store the configuration in a Secret.

Pod fails to start after update

edit

If you have configured a Beat to run as a Deployment and you are using a hostPath volume as the Beats data directory, you might encounter an error similar to the following:

ERROR   instance/beat.go:958    Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

This can happen if the new Pod is scheduled on the same Kubernetes node as the old Pod and is now trying to use the same data directory. Use a Recreate deployment strategy to avoid this problem.

« Configuration Examples Secure the Elastic Stack »