Debug standalone Elastic Agents
editDebug standalone Elastic Agents
editWhen you run standalone Elastic Agents, you are responsible for monitoring the status of your deployed Elastic Agents. You cannot view the status or logs in Fleet.
Use the following tips to help identify potential issues.
Also refer to Troubleshoot common problems for guidance on specific problems.
You might need to log in as a root user (or Administrator on Windows) to run these commands.
Check the status of the running Elastic Agent
editTo check the status of the running Elastic Agent daemon and other processes managed by
Elastic Agent, run the status
command. For example:
elastic-agent status
Returns something like:
State: HEALTHY Message: Running Fleet State: STOPPED Fleet Message: (no message) Components: * log (HEALTHY) Healthy: communicating with pid '25423' * filestream (HEALTHY) Healthy: communicating with pid '25424'
By default, this command returns the status in human-readable format. Use the
--output
flag to change it to json
or yaml
.
For more information about this command, refer to elastic-agent status.
Inspect Elastic Agent and related logs
editIf the Elastic Agent status is unhealthy, or behaving unexpectedly, inspect the logs of the running Elastic Agent.
The log location varies by platform. Elastic Agent logs are in the folders described
in Installation layout. Beats and Fleet Server logs are in folders named
for the output (for example, default
).
Start by investigating any errors you see in the Elastic Agent and related logs. Also look for repeated lines that might indicate problems like connection issues. If the Elastic Agent and related logs look clean, check the host operating system logs for out of memory (OOM) errors related to the Elastic Agent or any of its processes.
Increase the log level of the running Elastic Agent
editThe log level of the running agent is set to info
by default. At this level,
Elastic Agent will log informational messages, including the number of events that are
published. It also logs any warnings, errors, or critical errors.
To increase the log level, set it to debug
in the elastic-agent.yml
file.
The debug
setting configures Elastic Agent to log debug messages, including a
detailed printout of all flushed events, plus all the information collected at
other log levels.
Set other options if you want write logs to a file. For example:
agent.logging.level: debug agent.logging.to_files: true agent.logging.files: path: /var/log/elastic-agent name: elastic-agent keepfiles: 7 permissions: 0600
For other log settings, refer to Logging.
Expose /debug/pprof/ endpoints with the monitoring endpoint
editProfiling data produced by the /debug/pprof/
endpoints can be useful for
debugging, but presents a security risk. Do not expose these endpoints if the
monitoring endpoint is accessible over a network. (By default, the monitoring
endpoint is bound to a local Unix socket or Windows npipe and not accessible
over a network.)
To expose the /debug/pprof/
endpoints, set agent.monitoring.pprof: true
in
the elastic-agent.yml
file. For more information about monitoring settings,
refer to Monitoring.
After exposing the endpoints, you can access the HTTP handler bound to a socket for Beats or the Elastic Agent. For example:
sudo curl --unix-socket /Library/Elastic/Agent/data/tmp/default/filebeat/filebeat.sock http://socket/ | json_pp
Returns something like:
{ "beat" : "filebeat", "binary_arch" : "amd64", "build_commit" : "93708bd74e909e57ed5d9bea3cf2065f4cc43af3", "build_time" : "2022-01-28T09:53:29.000Z", "elastic_licensed" : true, "ephemeral_id" : "421e2525-9360-41db-9395-b9e627fbbe6e", "gid" : "0", "hostname" : "My-MacBook-Pro.local", "name" : "My-MacBook-Pro.local", "uid" : "0", "username" : "root", "uuid" : "fc0cc98b-b6d8-4eef-abf5-2d5f26adc7e8", "version" : "7.17.0" }
Likewise, the following request:
sudo curl --unix-socket /Library/Elastic/Agent/data/tmp/elastic-agent.sock http://socket/stats | json_pp
Returns something like:
{ "beat" : { "cpu" : { "system" : { "ticks" : 16272, "time" : { "ms" : 16273 } }, "total" : { "ticks" : 42981, "time" : { "ms" : 42982 }, "value" : 42981 }, "user" : { "ticks" : 26709, "time" : { "ms" : 26709 } } }, "info" : { "ephemeral_id" : "ea8fec0d-f7dd-4577-85d7-a2c38583c9c6", "uptime" : { "ms" : 5885653 }, "version" : "7.17.0" }, "memstats" : { "gc_next" : 13027776, "memory_alloc" : 7771632, "memory_sys" : 39666696, "memory_total" : 757970208, "rss" : 58990592 }, "runtime" : { "goroutines" : 101 } }, "system" : { "cpu" : { "cores" : 12 }, "load" : { "1" : 4.8892, "15" : 2.6748, "5" : 3.0537, "norm" : { "1" : 0.4074, "15" : 0.2229, "5" : 0.2545 } } } }
Inspect the Elastic Agent configuration
editTo inspect the running Elastic Agent configuration use the elastic-agent inspect command.
To analyze the current state of the agent, inspect log files, and see the configuration
of Elastic Agent and the sub-processes it starts, run the diagnostics
command. For example:
elastic-agent diagnostics
For more information about this command, refer to elastic-agent diagnostics.