Common problems

This section describes common problems you might encounter when using a Fleet-managed APM Server.

If no data shows up in Elasticsearch, first make sure that your APM components are properly connected.

apm-server test config
apm-server test output

The APM UI relies on index mappings to query and display data. If your APM data isn’t showing up in the APM UI, but is elsewhere in Kibana, like the Discover app, you may have a missing index mapping.

You can determine if a field was mapped correctly with the _mapping API. For example, run the following command in the Kibana console. This will display the field data type of the service.name field.

GET *apm*/_mapping/field/service.name

If the mapping.name.type is "text", your APM indices were not set up correctly.

".ds-metrics-apm.transaction.1m-default-2023.04.12-000038": {
   "mappings": {
      "service.name": {
         "full_name": "service.name",
         "mapping": {
            "name": {
               "type": "text" 
            }
         }
      }
   }
}

To fix this problem, install the APM integration by following these steps:

This will reinstall the APM index templates and trigger a data stream index rollover.

You can verify the correct index templates were installed by running the following command in the Kibana console:

GET /_index_template/traces-apm

The target host might be unreachable or the certificate may not be valid. To fix this problem:

This happens because your certificate is only valid for the hostname present in the Subject field. To resolve this problem, try one of these solutions:

This is not an SSL problem. It’s a networking problem. Make sure the two hosts can communicate.

This is not an SSL problem. Make sure that Logstash is running and that there is no firewall blocking the traffic.

A firewall is refusing the connection. Check if a firewall is blocking the traffic on the client, the network, or the destination host.

I/O Timeouts can occur when your timeout settings across the stack are not configured correctly, especially when using a load balancer.

You may see an error like the one below in the APM agent logs, and/or a similar error on the APM Server side:

[ElasticAPM] APM Server responded with an error:
"read tcp 123.34.22.313:8200->123.34.22.40:41602: i/o timeout"

To fix this, ensure timeouts are incrementing from the APM agent, through your load balancer, to the APM Server.

By default, the agent timeouts are set at 10 seconds, and the server timeout is set at 3600 seconds. Your load balancer should be set somewhere between these numbers.

For example:

APM agent --> Load Balancer  --> APM Server
   10s            15s               3600s

The APM Server timeout can be configured by updating the maximum duration for reading an entire request.

When adding too many distinct tag keys on a transaction or span, you risk creating a mapping explosion.

For example, you should avoid that user-specified data, like URL parameters, is used as a tag key. Likewise, using the current timestamp or a user ID as a tag key is not a good idea. However, tag values with a high cardinality are not a problem. Just try to keep the number of distinct tag keys at a minimum.

The symptom of a mapping explosion is that transactions and spans are not indexed anymore after a certain time. Usually, on the next day, the spans and transactions will be indexed again because a new index is created each day. But as soon as the field limit is reached, indexing stops again.

In the agent logs, you won’t see a sign of failures as the APM server asynchronously sends the data it received from the agents to Elasticsearch. However, the APM server and Elasticsearch log a warning like this:

{\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [1000] in [INDEX_NAME] has been exceeded\"}

Tail-based sampling requires minimal memory to run, and there should not be a noticeable increase in RSS memory usage. However, since tail-based sampling writes data to disk, it is possible to see a significant increase in OS page cache memory usage due to disk IO. If you see a drop in throughput and excessive disk activity after enabling tail-based sampling, please ensure that there is enough memory headroom in the system for OS page cache to perform disk IO efficiently.