Here are some approaches that you can use to diagnose the state of your Logstash and Kubernetes system, both in the event of any problems, and as part of a day-to-day approach to ensuring that everything is running as expected.

Checking resourcesedit

You can use the standard Kubernetes get and describe commands to quickly gather details about any resources in your Logstash and Kubernetes environment.

kubectl get pod logstash-7477d46bb7-4lcnv

NAME                                    READY   STATUS        RESTARTS         AGE
logstash-7477d46bb7-4lcnv               0/1     Pending       0                2m43s

If a Pod fails to reach the Running status after a few seconds, run this command to get more insights:

kubectl describe pod logstash-7477d46bb7-4lcnv

(...)
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  34s (x2 over 115s)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.

You can check the CPU and memory resources by running this command:

kubectl top pod logstash-7477d46bb7-4lcnv

NAME                        CPU(cores)   MEMORY(bytes)
logstash-7d5b749899-tfg4f   37m          882Mi

Viewing logsedit

Logstash Docker containers do not create log files by default. They log to standard output.

To view the log, run:

kubectl logs -f logstash-7477d46bb7-4lcnv

To enable debug log, set log.level: debug in logstash.yml in ConfigMap.

Connecting to a containeredit

At times, you may need to connect directly from your command shell into Logstash and other Kubernetes resources.

kubectl exec -it logstash-7477d46bb7-4lcnv -- bash

Running diagnosticsedit

Thread dumps and heap dumps can be helpful when you are debugging hard problems. Connect to the container, and then run the commands to gather the diagnostics.

jdk/bin/jstack -l 1 > /tmp/jstack_output.txt

jdk/bin/jcmd 1 GC.heap_dump /tmp/heap_dump.hprof

kubectl cp logstash-7477d46bb7-4lcnv:/tmp/heap_dump.hprof ./heap.hprof