Common problemsedit
Following are some suggested resolutions to problems that you may encounter when running Logstash in a Kubernetes environment.
Logstash keeps restartingedit
When you check the running Kubernetes pods status, Logstash shows continual restarts.
NAMESPACE NAME READY STATUS RESTARTS AGE default logstash-f7768c66d-grzbj 0/1 Running 3 (55s ago) 6m32s
This can be caused by a few issues:
Metrics API not accessible to readinessProbeedit
If the readinessProbe is unable to access the health check endpoint, the Logstash process will be continuously stopped and restarted. To fix it, set the following in logstash.yml in ConfigMap.
api.http.host: 0.0.0.0
Logstash startup process takes longer than initialDelaySecondsedit
Review the time constraints of readinessProbe and livenessProbe to ensure that Logstash has enough time to start up and expose the health check endpoint for the readiness and liveness probes to access.
Insufficient CPU or memory to start Logstashedit
Review CPU and memory usage using kubectl top pods (requires metrics server to be available for your Kubernetes implementation).
-
Set the values of
cpuandmemoryin yourDeploymentorStatefulSetappropriately. -
Ensure that the JVM memory settings are set appropriately. The default
Xmxvalue is1g, and we recommend that heap size is set to no more than 50-75% of total memory.
Logstash stops with OOM errorsedit
The status of Logstash shows Ready, but the pod repeatedly stops running.
This situation can be caused by insufficient memory. If Logstash uses more memory than the declared resource, Kubernetes shutdowns the pod immediately and the Logstash log does not show any shutdown related message.
Run kubectl get event --watch or kubectl describe pod if the event status shows OOMKilled.
The resolution is similar to the remedy for the insufficient CPU or memory problem. Review JVM and memory settings as shown in Insufficient CPU or memory to start Logstash.