IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Troubleshoot Frequently asked questions »

› ›

Troubleshoot common problems

edit

Troubleshoot common problems

edit

We have collected the most common known problems and listed them here. If your problem is not described here, please review the open issues in the following GitHub repositories:

Repo	To review or report issues about
elastic/kibana	Fleet and Integrations UI
elastic/beats	Elastic Agent or Beats shippers
elastic/fleet-server	Fleet Server
elastic/package-registry	Elastic Package Registry
elastic/observability-docs	Documentation issues

Have a question? Read our FAQ, or contact us in the discuss forum. Your feedback is valuable to us.

Running Elastic Agent standalone? Also refer to Debug standalone Elastic Agents.

Elastic Agent upgrade fails with `failed verification of agent binary` error

edit

There is a known issue in Elastic Agent version 8.1.1 and 8.1.2 that causes agent binary verification to fail with the following message:

[elastic_agent][error] failed to dispatch actions, error: failed verification of agent binary: 2 errors occurred:
fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-7f30bb/downloads/elastic-agent-<version>-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-7f30bb/downloads/elastic-agent-<version>-linux-x86_64.tar.gz.asc: no such file or directory
open /opt/Elastic/Agent/data/elastic-agent-7f30bb/downloads/elastic-agent-<version>-linux-x86_64.tar.gz.sha512: no such file or directory

For more information about resolving this problem, see Known issues.

Elastic Agents hosted on Elastic Cloud are stuck in `Updating` or `Offline`

edit

In Elastic Cloud, after upgrading Fleet Server and its integration policies, agents enrolled in the Elastic Cloud agent policy may experience issues updating. To resolve this problem:

Delete the Fleet Server integration policy under the Elastic Cloud agent policy:
1. To find the ID of the Fleet Server integration policy in Kibana, go to Management > Fleet > Agent Policies > Elastic Cloud agent policy > Fleet Server. This should take you to a URL like this:
  
  <kibana_ur>/app/fleet/policies/policy-elastic-agent-on-cloud/edit-integration/<integration_policy_id>
  
  Copy the policy ID at the end of the URL.
2. In a terminal window, run the following cURL request, providing your Kibana superuser credentials to delete the Fleet Server integration policy:
```
curl -u <username>:<password> --request POST \
  --url <kibana_url>/api/fleet/package_policies/delete \
  --header 'content-type: application/json' \
  --header 'kbn-xsrf: xyz' \
  --data '{"packagePolicyIds": ["<integration_policy_id>"], "force": true}' 
```
  <integration_policy_id> is the ID you copied in the previous step.
Remove existing policy references to the Elastic Agent cloud policy. In the Dev Tools Console, run:
```
POST .fleet-policies/_delete_by_query?q=policy-elastic-agent-on-cloud 
```
The user who runs this command must be authorized to delete restricted system indices.

Recreate the Fleet Server integration policy:

In a terminal window, run the following cURL request, providing your Kibana superuser credentials to force creation of the Fleet Server integration policy in the Elastic Cloud agent policy:

curl -u <username>:<password> --request POST \
  --url <kibana_url>/api/fleet/package_policies \
  --header 'content-type: application/json' \
  --header 'kbn-xsrf: xyz' \
  --data '{"name": "Fleet Server","description": "","namespace": "default","policy_id": "policy-elastic-agent-on-cloud","enabled": true,"output_id": "","inputs": [{"type": "fleet-server","policy_template": "fleet_server","enabled": true,"streams": [],"vars": {"host": {"value": ["0.0.0.0"],"type": "text"},"port": {"value": [8220],"type": "integer"},"max_connections": {"type": "integer"},"custom": {"value": "server.runtime:\n  gc_percent: 20","type": "yaml"}}}],"package": {"name": "fleet_server","title": "Fleet Server","version": "1.1.0"},"force": true}'

Force unenroll the agent stuck in Updating:

To find agent’s ID, go to Fleet > Agents and click the agent to see its details. Copy the Agent ID.

In a terminal window, run:

curl -u <username>:<password> --request POST \
  --url <kibana_url>/api/fleet/agents/<agentID>/unenroll \
  --header 'content-type: application/json' \
  --header 'kbn-xsrf: xx' \
  --data-raw '{"force":true,"revoke":true}' \
  --compressed

Where <agentID> is the ID you copied in the previous step.

Restart the APM & Fleet container:

In the Elastic Cloud console under APM & Fleet, click Force Restart.

When using Elastic Cloud, Fleet Server is not listed in Kibana

edit

If you are unable to see Fleet Server in Kibana, make sure it’s set up.

To set up Fleet Server on Elastic Cloud:

Go to your deployment on Elastic Cloud.
Follow the Elastic Cloud prompts to set up Integrations Server. Once complete, the Fleet Server Elastic Agent will show up in Fleet.

To enable Fleet and set up Fleet Server on a self-managed cluster:

In the Elasticsearch configuration file, config/elasticsearch.yml, set the following security settings to enable security and API keys:
```
xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true
```
In the Kibana configuration file, config/kibana.yml, enable Fleet and specify your user credentials:
```
xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters"
xpack.security.enabled: true
elasticsearch.username: "my_username" 
elasticsearch.password: "my_password"
```
Specify a user who is authorized to use Fleet.

To set up passwords, you can use the documented Elasticsearch APIs or the elasticsearch-setup-passwords command. For example, ./bin/elasticsearch-setup-passwords auto

After running the command:
1. Copy the Elastic user name to the Kibana configuration file.
2. Restart Kibana.
3. Follow the documented steps for setting up a self-managed Fleet Server. For more information, refer to Fleet Server.

The `/api/fleet/setup` endpoint can’t reach the package registry

edit

To install Integrations, the Fleet app requires a connection to an external service called the Elastic Package Registry.

For this to work, the Kibana server must connect to https://epr.elastic.co on port 443.

Fleet in Kibana crashes

edit

To investigate the error, open your browser’s development console.
Select the Network tab, and refresh the page.

One of the requests to the Fleet API will most likely have returned an error. If the error message doesn’t give you enough information to fix the problem, please contact us in the discuss forum.

Elastic Agent enrollment fails on the host with `x509: certificate signed by unknown authority` message

edit

To ensure that communication with Fleet Server is encrypted, Fleet Server requires Elastic Agents to present a signed certificate. In a self-managed cluster, if you don’t specify certificates when you set up Fleet Server, self-signed certificates are generated automatically.

If you attempt to enroll an Elastic Agent in a Fleet Server with a self-signed certificate, you will encounter the following error:

Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
Error: enroll command failed with exit code: 1

To fix this problem, pass the --insecure flag along with the enroll or install command. For example:

sudo ./elastic-agent install -f --url=https://<fleet-server-ip>:8220 --enrollment-token=<token> --insecure

Traffic between Elastic Agents and Fleet Server over HTTPS will be encrypted; you’re simply acknowledging that you understand that the certificate chain cannot be verified.

Allowing Fleet Server to generate self-signed certificates is useful to get things running for development, but not recommended in a production environment.

For more information, refer to Encrypt traffic in clusters with a self-managed Fleet Server.

Elastic Agent enrollment fails on the host with `x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs` message

edit

To ensure that communication with Elasticsearch is encrypted, Fleet Server requires Elasticsearch to present a signed certificate.

This error occurs when you use self-signed certificates with Elasticsearch using IP as a Common Name (CN). With IP as a CN, Fleet Server looks into subject alternative names (SANs), which is empty. To work around this situation, use the --fleet-server-es-insecure flag to disable certificate verification.

You will also need to set ssl.verification_mode: none in the Output settings in Fleet and Integrations UI.

Elastic Agent enrollment fails on the host with `Client.Timeout exceeded` message

edit

To enroll in Fleet, Elastic Agent must connect to the Fleet Server instance. If the agent is unable to connect, you see the following failure:

fail to enroll: fail to execute request to {fleet-server}:Post http://fleet-server:8220/api/fleet/agents/enroll?: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Here are several steps to help you troubleshoot the problem.

Check for networking problems. From the host, run the ping command to confirm that it can reach the Fleet Server instance.

Additionally, curl the /status api of Fleet Server:

curl -f http://<fleet-server-url>:8220/api/status

Verify that you have specified the correct Kibana Fleet settings URL and port for your environment.

By default, HTTPS protocol and port 8220 is expected by Fleet Server to communicate with Elasticsearch unless you have explicitly set it otherwise.
Check that you specified a valid enrollment key during enrollment. To do this:
1. In Fleet, select Enrollment tokens.
2. To view the secret, click the eyeball icon. The secret should match the string that you used to enroll Elastic Agent on your host.
3. If the secret doesn’t match, create a new enrollment token and use this token when you run the elastic-agent enroll command.

Many Fleet Server problems can be triaged and fixed with the below tips

edit

When creating an issue or sending a support forum communication, this section can help you identify what is required.

Fleet Server allows Elastic Agent to connect to Elasticsearch, which is the same as the connection to Kibana in prior releases. However, because Fleet Server is on the edge host, it may result in additional networking setup and troubleshooting.

Retrieve the Elastic Agent version

edit

If you installed the Elastic Agent, run the following command (the example is for POSIX based systems):
```
elastic-agent version
```
If you have not installed the Elastic Agent and you are running it as a temporary process, you can run:
```
./elastic-agent version
```
Both of the above commands are accessible via Windows or macOS with their OS-specific slight variation in how you call them. If needed, please refer to Install Elastic Agents for examples of how to adjust them.

Check the Elastic Agent status

edit

Run the following command to view the current status of the Elastic Agent.

elastic-agent status

Based on the information returned, you can take further action.

If Elastic Agent is running, but you do not see what you expect, here are some items to review:

In Fleet, click Agents. Check which policy is associated with the running Elastic Agent. If it is not the policy you expected, you can change it.
In Fleet, click Agents, and then select the Elastic Agent policy. Check for the integrations that should be included.

For example, if you want to include system data, make sure the System integration is included in the policy.
Confirm if the Collect agent logs and Collect agent metrics options are selected.
1. In Fleet, click Agents, and then select the Elastic Agent policy.
2. Select the Settings tab. If you want to collect agent logs or metrics, select these options.
  
  The Elastic Cloud agent policy is created only in Elastic Cloud deployments and, by default, does not include the collection of logs of metrics.

You can collect other files to assess and pass to the Elastic team for review for some more advanced debugging cases. These files should all be in the Elastic Agent directory on disk: state.yml, fleet.yml and elastic-agent.yml.

Collect Elastic Agent diagnostics bundle

edit

Run the following command to generate a zip archive containing diagnostics information that the Elastic team can use for debugging cases.

elastic-agent diagnostics collect

This archive collects the following information:

Elastic Agent versions numbers
Beats (and other process) version numbers and process metadata
Local configuration, elastic-agent policy, and the configuration that is rendered and passed to Beats and other processes
Elastic Agent’s local log files

Note that the diagnostics bundle is intended for debugging purposes only, its structure may change between releases.

The configuration files in the archive may have credentials in plain text. These will need to be manually redacted before the archive is shared.

The diagnostics command is also able to collect pprof data in the archive if the --pprof flag is passed:

elastic-agent diagnostics collect --pprof

By default this command takes a little more than 30s to run because it collects 30s trace and profile information from the Elastic Agent and all underlying Beats. To change the duration of the trace and profile data, set the --pprof-duration flag.

To set the timeout value for the collection command, use the --timeout flag. The timeout defaults to 30s + pprof-duration. Use this flag to increase the timeout if the diagnostics command times out before creating the archive.

Some problems occur so early that insufficient logging is available

edit

If some problems occur early and insufficient logging is available, run the following command:

./elastic-agent install -f

The stand-alone install command installs the Elastic Agent, and all of the service configuration is set up. You can now run the enrollment command. For example:

elastic-agent enroll -f --fleet-server-es=https://<es-url>:443 --fleet-server-service-token=<token> --fleet-server-policy=<policy-id>

Note: Port 443 is commonly used in Elastic Cloud. However, with self-managed deployments, your Elasticsearch may run on port 9200 or something entirely different.

For information on where to find agent logs, refer to our FAQ.

The Elastic Agent is cited as `Healthy` but still has set up problems sending data to Elasticsearch

edit

To confirm that the Elastic Agent is running and its status is Healthy, select the Agents tab.

If you previously selected the Collect agent logs option, you can now look at the agent logs.
Click the agent name and then select the Logs tab.

If there are no logs displayed, it suggests a communication problem between your host and Elasticsearch. The possible reason for this is that the port is already in use.
You can check the port usage using tools like Wireshark or netstat. On a POSIX system, you can run the following command:
```
netstat -nat | grep :8220
```
Any response data indicates that the port is in use. This could be correct or not if you had intended to uninstall the Fleet Server. In which case, re-check and continue.

Elastic Agent is stuck in status `Updating`

edit

In some rare situations during the upgrade process, an Elastic Agent may hang with the status Updating. To fix this problem, authenticate as a superuser and force unenroll to remove the stale Elastic Agent:

curl 'https://<kibana_url>/api/fleet/agents/<agentID>/unenroll' \
  -H 'kbn-xsrf: xx' \
  -H 'Content-type: application/json' \
  --data-raw '{"force":true,"revoke":true}' \
  --compressed

Fleet Server is running and healthy with data, but other Agents cannot use it to connect to Elasticsearch

edit

Some settings are only used when you have multiple Elastic Agents. If this is the case, it may help to check that the hosts can communicate with the Fleet Server.

From the non-Fleet Server host, run the following command:

curl -f http://<fleet-server-ip>:8220/api/status

The response may yield errors that you can be debug further, or it may work and show that communication ports and networking are not the problems.

One common problem is that the default Fleet Server port of 8220 isn’t open on the Fleet Server host to communicate. You can review and correct this using common tools in alignment with any networking and security concerns you may have.

Elasticsearch authentication service fails with `Authentication using apikey failed` message

edit

To save API keys and encrypt them in Elasticsearch, Fleet requires an encryption key.

To provide an API key, in the kibana.yml configuration file, set the xpack.encryptedSavedObjects.encryptionKey property.

xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters"

Elastic Agent fails with `Agent process is not root/admin or validation failed` message

edit

Ensure the user running Elastic Agent has root privileges as some integrations require root privileges to collect sensitive data.

If you’re running Elastic Agent in the foreground (and not as a service) on Linux or macOS, run the agent under the root user: sudo or su.

If you’re using the Endpoint Security integration, make sure you’re running Elastic Agent under the SYSTEM account.

If you install Elastic Agent as a service as described in Install Elastic Agents, Elastic Agent runs under the SYSTEM account by default.

To run Elastic Agent under the SYSTEM account, you can do the following:

Download PsExec and extract the contents to a folder. For example, d:\tools.
Open a command prompt as an Administrator (right-click the command prompt icon and select Run As Administrator).

From the command prompt, run Elastic Agent under the SYSTEM account:

d:\tools\psexec.exe -sid "C:\Program Files\Elastic-Agent\elastic-agent.exe" run

Integration policy upgrade has too many conflicts

edit

If you try to upgrade an integration policy that is several versions old, there may be substantial conflicts or configuration issues. Rather than trying to fix these problems, it might be faster to create a new policy, test it, and roll out the integration upgrade to additional hosts.

After upgrading the integration:

Create a new policy.
Add the integration to the policy. The newer version is automatically used.
Apply the policy to an Elastic Agent.

In larger deployments, you should test integration upgrades on a sample Elastic Agent before rolling out a larger upgrade initiative. Only after a small trial is deemed successful should the updated policy be rolled out all hosts.
Roll out the integration update to additional hosts:
1. In Fleet, click Agent policies. Click on the name of the policy you want to edit.
2. Search or scroll to a specific integration. Open the Actions menu and select Delete integration.
3. Click Add integration and re-add the freshly deleted integration. The updated version will be used and applied to all Elastic Agents.
4. Repeat this process for each policy with the out-of-date integration.
  
  In some instances, for example, when there are hundreds or thousands of different Elastic Agents and policies that need to be updated, this upgrade path is not feasible. In this case, update one policy and use the Copy a policy action to apply the updated policy versions to additional policies. This method’s downside is losing the granularity of assessing the individual Integration version changes individually across policies.

Elastic Agent hangs while unenrolling

edit

When unenrolling Elastic Agent, Fleet waits for acknowledgement from the agent before it completes the unenroll process. If Fleet doesn’t receive an acknowledgement, the status hangs at unenrolling.

You can unenroll an agent to invalidate all API keys related to the agent and change the status to inactive so that the agent no longer appears in Fleet.

In Fleet, select Agents.
Under Agents, choose Unenroll agent from the Actions menu next to the agent you want to unenroll.
Click Force unenroll.

On Fleet Server startup, ERROR seen with `State changed to CRASHED: exited with code: 1`

edit

You may see this error message for a number of different reasons. A common reason is when attempting production-like usage and the ca.crt file passed in cannot be found. To verify if this is the problem, bootstrap Fleet Server without passing a ca.crt file. This implies you would test any subsequent Elastic Agent installs temporarily with {fleet-sever}'s own self-signed cert.

Ensure to pass in the full path to the ca.crt file. A relative path is not viable.

You will know if your Fleet Server is set up with its testing oriented self-signed certificate usage, when you see the following error during Elastic Agent installs:

Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
Error: enroll command failed with exit code: 1

To install or enroll against a self-signed cert Fleet Server Elastic Agent, add in the --insecure option to the command:

sudo ./elastic-agent install -f --url=https://<fleet-server-ip>:8220 --enrollment-token=<token> --insecure

For more information, refer to Elastic Agent enrollment fails on the host with x509: certificate signed by unknown authority message.

Elastic Endpoint uninstallation fails

edit

When you uninstall Elastic Agent, all the programs managed by Elastic Agent, such as Elastic Endpoint, are also removed. If uninstallation fails, Elastic Endpoint might remain on your system.

To remove Elastic Endpoint, run the following commands:

cd /tmp
cp /Library/Elastic/Endpoint/elastic-endpoint elastic-endpoint
sudo ./elastic-endpoint uninstall
rm elastic-endpoint

cd /tmp
cp /opt/Elastic/Endpoint/elastic-endpoint elastic-endpoint
sudo ./elastic-endpoint uninstall
rm elastic-endpoint

cd %TEMP%
copy "c:\Program Files\Elastic\Endpoint\elastic-endpoint.exe" elastic-endpoint.exe
.\elastic-endpoint.exe uninstall
del .\elastic-endpoint.exe

API key is unauthorized to send telemetry to `.logs-endpoint.diagnostic.collection-*` indices

edit

By default, telemetry is turned on in the Elastic Stack to helps us learn about the features that our users are most interested in. This helps us to focus our efforts on making features even better.

If you’ve recently upgraded from version 7.10 to 7.11, you might see the following message when you view Endpoint Security logs:

action [indices:admin/auto_create] is unauthorized for API key id [KbvCi3YB96EBa6C9k2Cm]
of user [fleet_enroll] on indices [.logs-endpoint.diagnostic.collection-default]

The above message indicates that Elastic Endpoint does not have the correct permissions to send telemetry. This is a known problem in 7.11 that will be fixed in an upcoming patch release.

To remove this message from your logs, you can turn off telemetry for the Endpoint Security integration until the next patch release is available.

In Kibana, click Integrations, and then select the Manage tab.
Click Endpoint Security, and then select the Policies tab to view all the installed integrations.
Click the integration to edit it.
Under advanced settings, set windows.advanced.diagnostic.enabled to false, and then save the integration.

Hosted Elastic Agent is offline

edit

To scale the Fleet Server deployment, Elastic Cloud starts new containers or shuts down old ones when hosted Elastic Agents are required or no longer needed. The old Elastic Agents will show in the Agents list for 24 hours then automatically disappear.

APM & Fleet fails to upgrade to 8.x on Elastic Cloud

edit

In some scenarios, upgrading APM & Fleet to 8.x may fail if the Elastic Cloud agent policy was modified manually. The Fleet app in Kibana may show a message like:

Unable to create package policy. Package 'apm' already exists on this agent policy

To work around this problem, you can reset the Elastic Cloud agent policy with an API call. Note that this will remove any custom integration policies that you’ve added to the policy, such as Synthetics monitors.

curl -u elastic:<password> --request POST \
  --url <kibana_url>/internal/fleet/reset_preconfigured_agent_policies/policy-elastic-agent-on-cloud \
  --header 'Content-Type: application/json' \
  --header 'kbn-xsrf: xyz' \

« Troubleshoot Frequently asked questions »

Troubleshoot common problems

Troubleshoot common problems

Elastic Agent upgrade fails with failed verification of agent binary error

Elastic Agents hosted on Elastic Cloud are stuck in Updating or Offline

When using Elastic Cloud, Fleet Server is not listed in Kibana

The /api/fleet/setup endpoint can’t reach the package registry

Fleet in Kibana crashes

Elastic Agent enrollment fails on the host with x509: certificate signed by unknown authority message

Elastic Agent enrollment fails on the host with x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs message

Elastic Agent enrollment fails on the host with Client.Timeout exceeded message

Many Fleet Server problems can be triaged and fixed with the below tips

Retrieve the Elastic Agent version

Check the Elastic Agent status

Collect Elastic Agent diagnostics bundle

Some problems occur so early that insufficient logging is available

The Elastic Agent is cited as Healthy but still has set up problems sending data to Elasticsearch

Elastic Agent is stuck in status Updating

Fleet Server is running and healthy with data, but other Agents cannot use it to connect to Elasticsearch

Elasticsearch authentication service fails with Authentication using apikey failed message

Elastic Agent fails with Agent process is not root/admin or validation failed message

Integration policy upgrade has too many conflicts

Elastic Agent hangs while unenrolling

On Fleet Server startup, ERROR seen with State changed to CRASHED: exited with code: 1

Elastic Endpoint uninstallation fails

API key is unauthorized to send telemetry to .logs-endpoint.diagnostic.collection-* indices

Hosted Elastic Agent is offline

APM & Fleet fails to upgrade to 8.x on Elastic Cloud

Elastic Agent upgrade fails with `failed verification of agent binary` error

Elastic Agents hosted on Elastic Cloud are stuck in `Updating` or `Offline`

The `/api/fleet/setup` endpoint can’t reach the package registry

Elastic Agent enrollment fails on the host with `x509: certificate signed by unknown authority` message

Elastic Agent enrollment fails on the host with `x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs` message

Elastic Agent enrollment fails on the host with `Client.Timeout exceeded` message

The Elastic Agent is cited as `Healthy` but still has set up problems sending data to Elasticsearch

Elastic Agent is stuck in status `Updating`

Elasticsearch authentication service fails with `Authentication using apikey failed` message

Elastic Agent fails with `Agent process is not root/admin or validation failed` message

On Fleet Server startup, ERROR seen with `State changed to CRASHED: exited with code: 1`

API key is unauthorized to send telemetry to `.logs-endpoint.diagnostic.collection-*` indices