Troubleshoot common problems

We have collected the most common known problems and listed them here. If your problem is not described here, please review the open issues in the following GitHub repositories:

Have a question? Read our FAQ, or contact us in the discuss forum. Your feedback is valuable to us.

Running Elastic Agent standalone? Also refer to Debug standalone Elastic Agents.

Find troubleshooting information for Fleet, Fleet Server, and Elastic Agent in the following documentation:

In Fleet, if you delete an Elastic Agent policy that is associated with one or more inactive enrolled agents, when the agent returns back to a Healthy or Offline state, it cannot be unenrolled. Attempting to unenroll the agent results in an Error unenrolling agent message, and the unenrollment fails.

To resolve this problem, you can use the Kibana Fleet APIs to force unenroll the agent.

To uninstall a single Elastic Agent:

POST kbn:/api/fleet/agents/<agent_id>/unenroll
{
  "force": true,
  "revoke": true
}

To bulk uninstall a set of Elastic Agents:

POST kbn:/api/fleet/agents/bulk_unenroll
{ "agents": ["<agent_id1>", "<agent-id2>"],
  "force": true,
  "revoke": true
}

We are also updating the Fleet UI to prevent removal of an Elastic Agent policy that is currently associated with any inactive agents.

When you use an Elastic Agent integration in which TSDB (Time Series Database) is enabled, you may encounter an illegal_argument_exception error in the Fleet UI.

This can occur if you have a component template defined that includes a _source attribute, which conflicts with the _source: synthetic setting used when TSDB is enabled.

For details about the error and how to resolve it, refer to the section Runtime fields cannot be used in TSDB indices in the Innovation Hub article TSDB enabled integrations for Elastic Agent.

In Elastic Cloud, after upgrading Fleet Server and its integration policies, agents enrolled in the Elastic Cloud agent policy may experience issues updating. To resolve this problem:

If you are unable to see Fleet Server in Kibana, make sure it’s set up.

To set up Fleet Server on Elastic Cloud:

To enable Fleet and set up Fleet Server on a self-managed cluster:

To install Integrations, the Fleet app requires a connection to an external service called the Elastic Package Registry.

For this to work, the Kibana server must connect to https://epr.elastic.co on port 443.

In air-gapped environments, you may encounter the following error if you’re using a custom Certificate Authority (CA) that is not available to Kibana:

{"type":"log","@timestamp":"2022-03-02T09:58:36-05:00","tags":["error","plugins","fleet"],"pid":58716,"message":"Error connecting to package registry: request to https://customer.server.name:8443/categories?experimental=true&include_policy_templates=true&kibana.version=7.17.0 failed, reason: self signed certificate in certificate chain"}

To fix this problem, add your CA certificate file path to the Kibana startup file by defining the NODE_EXTRA_CA_CERTS environment variable. More information about this in TLS configuration of the Elastic Package Registry section.

To ensure that communication with Fleet Server is encrypted, Fleet Server requires Elastic Agents to present a signed certificate. In a self-managed cluster, if you don’t specify certificates when you set up Fleet Server, self-signed certificates are generated automatically.

If you attempt to enroll an Elastic Agent in a Fleet Server with a self-signed certificate, you will encounter the following error:

Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
Error: enroll command failed with exit code: 1

To fix this problem, pass the --insecure flag along with the enroll or install command. For example:

sudo ./elastic-agent install --url=https://<fleet-server-ip>:8220 --enrollment-token=<token> --insecure

Traffic between Elastic Agents and Fleet Server over HTTPS will be encrypted; you’re simply acknowledging that you understand that the certificate chain cannot be verified.

Allowing Fleet Server to generate self-signed certificates is useful to get things running for development, but not recommended in a production environment.

For more information, refer to Configure SSL/TLS for self-managed Fleet Servers.

To ensure that communication with Elasticsearch is encrypted, Fleet Server requires Elasticsearch to present a signed certificate.

This error occurs when you use self-signed certificates with Elasticsearch using IP as a Common Name (CN). With IP as a CN, Fleet Server looks into subject alternative names (SANs), which is empty. To work around this situation, use the --fleet-server-es-insecure flag to disable certificate verification.

You will also need to set ssl.verification_mode: none in the Output settings in Fleet and Integrations UI.

To enroll in Fleet, Elastic Agent must connect to the Fleet Server instance. If the agent is unable to connect, you see the following failure:

fail to enroll: fail to execute request to {fleet-server}:Post http://fleet-server:8220/api/fleet/agents/enroll?: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Here are several steps to help you troubleshoot the problem.

Fleet Server allows Elastic Agent to connect to Elasticsearch, which is the same as the connection to Kibana in prior releases. However, because Fleet Server is on the edge host, it may result in additional networking setup and troubleshooting.

Run the following command to view the current status of the Elastic Agent.

elastic-agent status

Based on the information returned, you can take further action.

If Elastic Agent is running, but you do not see what you expect, here are some items to review:

The Elastic Agent diagnostics bundle collects the following information:

Note that the diagnostics bundle is intended for debugging purposes only, its structure may change between releases.

Get the diagnostics bundle using the CLI

Run the following command to generate a zip archive containing diagnostics information that the Elastic team can use for debugging cases.

elastic-agent diagnostics

If you want to omit the raw events from the diagnostic, add the flag --exclude-events.

Get the diagnostics bundle through Fleet

Fleet provides the ability to remotely generate and gather an Elastic Agent’s diagnostics bundle. An agent can gather and upload diagnostics if it is online in a Healthy or Unhealthy state. The diagnostics are sent to Fleet Server which in turn adds it into Elasticsearch. Therefore, this works even with Elastic Agents that are not using the Elasticsearch output. To download the diagnostics bundle for local viewing:

Any in-progress or previously collected bundles for the Elastic Agent will be listed on this page.

Note that the bundles are stored in Elasticsearch and are removed automatically after 7 days. You can also delete any previously created bundle by clicking the trash can icon.

If some problems occur early and insufficient logging is available, run the following command:

./elastic-agent install -f

The stand-alone install command installs the Elastic Agent, and all of the service configuration is set up. You can now run the enrollment command. For example:

elastic-agent enroll --fleet-server-es=https://<es-url>:443 --fleet-server-service-token=<token> --fleet-server-policy=<policy-id>

Note: Port 443 is commonly used in Elastic Cloud. However, with self-managed deployments, your Elasticsearch may run on port 9200 or something entirely different.

For information on where to find agent logs, refer to our FAQ.

Beginning in Elastic Stack version 8.11, a stuck Elastic Agent upgrade should be detected automatically, and you can restart the upgrade from Fleet.

Some settings are only used when you have multiple Elastic Agents. If this is the case, it may help to check that the hosts can communicate with the Fleet Server.

From the non-Fleet Server host, run the following command:

curl -f http://<fleet-server-ip>:8220/api/status

The response may yield errors that you can be debug further, or it may work and show that communication ports and networking are not the problems.

One common problem is that the default Fleet Server port of 8220 isn’t open on the Fleet Server host to communicate. You can review and correct this using common tools in alignment with any networking and security concerns you may have.

To save API keys and encrypt them in Elasticsearch, Fleet requires an encryption key.

To provide an API key, in the kibana.yml configuration file, set the xpack.encryptedSavedObjects.encryptionKey property.

xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters"

Ensure the user running Elastic Agent has root privileges as some integrations require root privileges to collect sensitive data.

If you’re running Elastic Agent in the foreground (and not as a service) on Linux or macOS, run the agent under the root user: sudo or su.

If you’re using the Elastic Defend integration, make sure you’re running Elastic Agent under the SYSTEM account.

To run Elastic Agent under the SYSTEM account, you can do the following:

If you try to upgrade an integration policy that is several versions old, there may be substantial conflicts or configuration issues. Rather than trying to fix these problems, it might be faster to create a new policy, test it, and roll out the integration upgrade to additional hosts.

After upgrading the integration:

When unenrolling Elastic Agent, Fleet waits for acknowledgment from the agent before it completes the unenroll process. If Fleet doesn’t receive an acknowledgment, the status hangs at unenrolling.

You can unenroll an agent to invalidate all API keys related to the agent and change the status to inactive so that the agent no longer appears in Fleet.

You may see this error message for a number of different reasons. A common reason is when attempting production-like usage and the ca.crt file passed in cannot be found. To verify if this is the problem, bootstrap Fleet Server without passing a ca.crt file. This implies you would test any subsequent Elastic Agent installs temporarily with {fleet-sever}'s own self-signed cert.

You will know if your Fleet Server is set up with its testing oriented self-signed certificate usage, when you see the following error during Elastic Agent installs:

Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
Error: enroll command failed with exit code: 1

To install or enroll against a self-signed cert Fleet Server Elastic Agent, add in the --insecure option to the command:

sudo ./elastic-agent install --url=https://<fleet-server-ip>:8220 --enrollment-token=<token> --insecure

For more information, refer to Elastic Agent enrollment fails on the host with x509: certificate signed by unknown authority message.

When you uninstall Elastic Agent, all the programs managed by Elastic Agent, such as Elastic Endpoint, are also removed. If uninstalling fails, Elastic Endpoint might remain on your system.

To remove Elastic Endpoint, run the following commands:

cd /tmp
cp /Library/Elastic/Endpoint/elastic-endpoint elastic-endpoint
sudo ./elastic-endpoint uninstall
rm elastic-endpoint

cd /tmp
cp /opt/Elastic/Endpoint/elastic-endpoint elastic-endpoint
sudo ./elastic-endpoint uninstall
rm elastic-endpoint

cd %TEMP%
copy "c:\Program Files\Elastic\Endpoint\elastic-endpoint.exe" elastic-endpoint.exe
.\elastic-endpoint.exe uninstall
del .\elastic-endpoint.exe

By default, telemetry is turned on in the Elastic Stack to helps us learn about the features that our users are most interested in. This helps us to focus our efforts on making features even better.

If you’ve recently upgraded from version 7.10 to 7.11, you might see the following message when you view Elastic Defend logs:

action [indices:admin/auto_create] is unauthorized for API key id [KbvCi3YB96EBa6C9k2Cm]
of user [fleet_enroll] on indices [.logs-endpoint.diagnostic.collection-default]

The above message indicates that Elastic Endpoint does not have the correct permissions to send telemetry. This is a known problem in 7.11 that will be fixed in an upcoming patch release.

To remove this message from your logs, you can turn off telemetry for the Elastic Defend integration until the next patch release is available.

To scale the Fleet Server deployment, Elastic Cloud starts new containers or shuts down old ones when hosted Elastic Agents are required or no longer needed. The old Elastic Agents will show in the Agents list for 24 hours then automatically disappear.

If you’re testing Fleet Server locally on a macOS system using localhost (https://127.0.0.1:8220) as the Host URL, you may encounter this error:

Error: fail to enroll: fail to execute request to fleet-server:
lookup My-MacBook-Pro.local: no such host

This can occur on newer macOS software. To resolve the problem, ensure that file sharing is enabled on your local system.

In some scenarios, upgrading APM & Fleet to 8.x may fail if the Elastic Cloud agent policy was modified manually. The Fleet app in Kibana may show a message like:

Unable to create package policy. Package 'apm' already exists on this agent policy

To work around this problem, you can reset the Elastic Cloud agent policy with an API call. Note that this will remove any custom integration policies that you’ve added to the policy, such as Synthetics monitors.

curl -u elastic:<password> --request POST \
  --url <kibana_url>/internal/fleet/reset_preconfigured_agent_policies/policy-elastic-agent-on-cloud \
  --header 'Content-Type: application/json' \
  --header 'kbn-xsrf: xyz'

In versions 8.9 and above, an Elastic Agent upgrade may fail when the upgrader can’t access a PGP key required to verify the binary signature. For details and a workaround, refer to the PGP key download fails in an air-gapped environment known issue in the version 8.9.0 Release Notes or to the workaround documentation in the elastic-agent GitHub repository.

When you use Fleet-managed Elastic Agent, at least one Elastic Agent needs to be running the Fleet Server integration. In case the policy containing this integration is accidentally removed from Elastic Agent, all other agents will not be able to be managed. However, the Elastic Agents will continue to send data to their configured output.

There are two approaches to fixing this issue, depending on whether or not the the Elastic Agent that was running the Fleet Server integration is still installed and healthy (but is now running another policy).

To recover the Elastic Agent:

After running these steps your Elastic Agents should be able to connect with Fleet again.

In a Kubernetes environment, Elastic Agent may be terminated with reason OOMKilled due to inadequate available memory.

To detect the problem, run the kubectl describe pod command and check the results for the following content:

       Last State:   Terminated
       Reason:       OOMKilled
       Exit Code:    137

To resolve the problem, allocate additional memory to the agent and then restart it.

On Linux systems, when you install Elastic Agent without administrative privileges, that is, using the --unprivileged flag, Elastic Agent commands should not be run with sudo. Doing so may result in an error due to the agent not having the required privileges.

For example, when you run Elastic Agent with the --unprivileged flag, running the elastic-agent inspect command will result in an error like the following:

Error: error loading agent config: error loading raw config: fail to read configuration /Library/Elastic/Agent/fleet.enc for the elastic-agent: fail to decode bytes: cipher: message authentication failed

To resolve this, either install Elastic Agent without the --unprivileged flag so that it has administrative access, or run the Elastic Agent commands without the sudo prefix.

Potential issues during Elastic Agent installation on Kubernetes can be categorized into two main areas:

When troubleshooting installations performed with Kustomize, it’s good practice to inspect the output of the rendered manifest. To do this, take the installation command provided by Kibana Onboarding and replace the final part, | kubectl apply -f-, with a redirection to a local file. This allows for easier analysis of the rendered output.

For example, the following command, originally provided by Kibana for an Elastic Agent Standalone installation, has been modified to redirect the output for troubleshooting purposes:

kubectl kustomize https://github.com/elastic/elastic-agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone\?ref\=v8.15.3 | sed -e 's/JUFQSV9LRVkl/ZDAyNnZaSUJ3eWIwSUlCT0duRGs6Q1JfYmJoVFRUQktoN2dXTkd0FNMtdw==/g' -e "s/%ES_HOST%/https:\/\/7a912e8674a34086eacd0e3d615e6048.us-west2.gcp.elastic-cloud.com:443/g" -e "s/%ONBOARDING_ID%/db687358-2c1f-4ec9-86e0-8f1baa4912ed/g" -e "s/\(docker.elastic.co\/beats\/elastic-agent:\).*$/\18.15.3/g" -e "/{CA_TRUSTED}/c\ " > elastic_agent_installation_complete_manifest.yaml

The previous command generates a local file named elastic_agent_installation_complete_manifest.yaml, which you can use for further analysis. It contains the complete set of resources required for the Elastic Agent installation, including:

The content of this file is equivalent to what you’d obtain by following the Run Elastic Agent Standalone on Kubernetes steps, with the exception that kube-state-metrics is not included in the standalone method.

Possible issues

If the installation is correct and all resources are deployed, but data is not flowing as expected (for example, you don’t see any data on the [Metrics Kubernetes] Cluster Overview dashboard), check the following items:

Possible issues

Extra resources for Elastic Agent on Kubernetes troubleshooting and information:

If an agent was unenrolled from a Kubernetes cluster, there might be data remaining in /var/lib/elastic-agent-managed/kube-system/state on the node(s). Reenrolling an agent later on the same nodes might then result in invalid api key to authenticate with fleet error messages.

To avoid these errors, make sure to delete this state-folder before enrolling a new agent.

For more information, refer to issue #3586.