Debug standalone Elastic Agents

When you run standalone Elastic Agents, you are responsible for monitoring the status of your deployed Elastic Agents. You cannot view the status or logs in Fleet.

Use the following tips to help identify potential issues.

Also refer to Troubleshoot common problems for guidance on specific problems.

To check the status of the running Elastic Agent daemon and other processes managed by Elastic Agent, run the status command. For example:

elastic-agent status

Returns something like:

Status: HEALTHY
Message: (no message)
Applications:
  * metricbeat  (HEALTHY)
                Running
  * filebeat    (STOPPING)
                (no message)

By default, this command returns the status in human-readable format. Use the --output flag to change it to json or yaml.

For more information about this command, refer to elastic-agent status.

If the Elastic Agent status is unhealthy, or behaving unexpectedly, inspect the logs of the running Elastic Agent.

The log location varies by platform. Elastic Agent logs are in the folders described in Installation layout. Beats and Fleet Server logs are in folders named for the output (for example, default). Elastic Endpoint logs are in the installation directory.

Start by investigating any errors you see in the Elastic Agent and related logs. Also look for repeated lines that might indicate problems like connection issues. If the Elastic Agent and related logs look clean, check the host operating system logs for out of memory (OOM) errors related to the Elastic Agent or any of its processes.

The log level of the running agent is set to info by default. At this level, Elastic Agent will log informational messages, including the number of events that are published. It also logs any warnings, errors, or critical errors.

To increase the log level, set it to debug in the elastic-agent.yml file.

The debug setting configures Elastic Agent to log debug messages, including a detailed printout of all flushed events, plus all the information collected at other log levels.

Set other options if you want write logs to a file. For example:

agent.logging.level: debug
agent.logging.to_files: true
agent.logging.files:
  path: /var/log/elastic-agent
  name: elastic-agent
  keepfiles: 7
  permissions: 0600

For other log settings, refer to Logging.

Profiling data produced by the /debug/pprof/ endpoints can be useful for debugging, but presents a security risk. Do not expose these endpoints if the monitoring endpoint is accessible over a network. (By default, the monitoring endpoint is bound to a local unix socket or Windows npipe and not accessible over a network.)

To expose the /debug/pprof/ endpoints, set agent.monitoring.pprof: true in the elastic-agent.yml file. For more information about monitoring settings, refer to Monitoring.

After exposing the endpoints, you can access the http handler bound to a socket for Beats or the Elastic Agent. For example:

sudo curl --unix-socket /Library/Elastic/Agent/data/tmp/default/filebeat/filebeat.sock http://socket/ | json_pp

Returns something like:

{
   "beat" : "filebeat",
   "binary_arch" : "amd64",
   "build_commit" : "93708bd74e909e57ed5d9bea3cf2065f4cc43af3",
   "build_time" : "2022-01-28T09:53:29.000Z",
   "elastic_licensed" : true,
   "ephemeral_id" : "421e2525-9360-41db-9395-b9e627fbbe6e",
   "gid" : "0",
   "hostname" : "My-MacBook-Pro.local",
   "name" : "My-MacBook-Pro.local",
   "uid" : "0",
   "username" : "root",
   "uuid" : "fc0cc98b-b6d8-4eef-abf5-2d5f26adc7e8",
   "version" : "7.17.0"
}

Likewise, the following request:

sudo curl --unix-socket /Library/Elastic/Agent/data/tmp/elastic-agent.sock http://socket/stats | json_pp

Returns something like:

{
   "beat" : {
      "cpu" : {
         "system" : {
            "ticks" : 16272,
            "time" : {
               "ms" : 16273
            }
         },
         "total" : {
            "ticks" : 42981,
            "time" : {
               "ms" : 42982
            },
            "value" : 42981
         },
         "user" : {
            "ticks" : 26709,
            "time" : {
               "ms" : 26709
            }
         }
      },
      "info" : {
         "ephemeral_id" : "ea8fec0d-f7dd-4577-85d7-a2c38583c9c6",
         "uptime" : {
            "ms" : 5885653
         },
         "version" : "7.17.0"
      },
      "memstats" : {
         "gc_next" : 13027776,
         "memory_alloc" : 7771632,
         "memory_sys" : 39666696,
         "memory_total" : 757970208,
         "rss" : 58990592
      },
      "runtime" : {
         "goroutines" : 101
      }
   },
   "system" : {
      "cpu" : {
         "cores" : 12
      },
      "load" : {
         "1" : 4.8892,
         "15" : 2.6748,
         "5" : 3.0537,
         "norm" : {
            "1" : 0.4074,
            "15" : 0.2229,
            "5" : 0.2545
         }
      }
   }
}

To inspect the Elastic Agent configuration, run the inspect command. For example:

elastic-agent inspect

Use the --output flag to inspect the configuration passed to other processes, such as Filebeat. For example:

elastic-agent inspect output --output default --program filebeat

Returns something like:

[default] filebeat:
filebeat:
  inputs:
  - exclude_files:
    - .gz$
    id: logfile-system.auth-default-system
    index: logs-system.auth-default
    meta:
      package:
        name: system
        version: 7.16.3
    multiline:
      match: after
      pattern: ^\s
    name: system-1
    paths:
    - /var/log/auth.log*
    - /var/log/secure*
    processors:
    - add_locale: null
    - add_fields:
        fields:
          dataset: system.auth
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: system.auth
        target: event
    - add_fields:
        fields:
          id: 3c4a8f14-561a-449f-8935-7485cd494bac
          snapshot: false
          version: 7.16.3
        target: elastic_agent
    - add_fields:
        fields:
          id: 3c4a8f14-561a-449f-8935-7485cd494bac
        target: agent
    revision: 1
    type: log
  - exclude_files:
    - .gz$
    id: logfile-system.syslog-default-system
    index: logs-system.syslog-default
    meta:
      package:
        name: system
        version: 1.6.4
    multiline:
      match: after
      pattern: ^\s
    name: system-1
    paths:
    - /var/log/messages*
    - /var/log/syslog*
    processors:
    - add_locale: null
    - add_fields:
        fields:
          dataset: system.syslog
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: system.syslog
        target: event
    - add_fields:
        fields:
          id: 3c4a8f14-561a-449f-8935-7485cd494bac
          snapshot: false
          version: 7.16.3
        target: elastic_agent
    - add_fields:
        fields:
          id: 3c4a8f14-561a-449f-8935-7485cd494bac
        target: agent
    revision: 1
    type: log
output:
  elasticsearch:
    api_key: your:apikey
    hosts:
    - https://5d87573b66ed4d7f6cd1d2d3f1e30bc5.us-central1.gcp.foundit.no:443

For more information about this command, refer to elastic-agent inspect.