Troubleshooting

Something isn’t quite working as expected? Here are some guidelines how to find out what’s going wrong.

As a first step, please check if your stack is compatible with the currently supported technologies.

Don’t worry if you can’t figure out what the problem is. Open a topic in the APM discuss forum and we will help you out.

Agent is updated frequently and releases are not strongly tied to other components in the stack.

Therefore, trying to update to the most recently released agent version is often the recommended first troubleshooting step. If possible, trying the latest-snapshot is even preferable as it would include fixes that are not yet released.

See upgrading documentation for more details.

Like many other Java agents, our agent instruments classes by injecting bytecode into them on runtime. Our bytecode instrumentation guarantees to produce only valid bytecode and to never change the class’s "schema". If you are using other Java agents in addition to ours that have the same guarantees, they should not interfere with each other. However, some Java agents do not conform to these guarantees, in which case there may be issues with using them in parallel to ours. If you encounter errors, one of the first things to do is to remove any additional agent in order to check whether this is indeed a case of agent mismatch. We may still find a way to make them work together, but this information would be crucial for our ability to assist, so make sure to include it when reporting such issues.

There are several logging related configuration options. The most important one is log_level.

Set the log level to DEBUG or even TRACE to get more information about the behavior of the agent.

Please always post the whole content of your log files when asking for help. Use the procedure to ensure consistent logs when reporting potential issues.

When the agent starts up, you should see logs similar to these:

[main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM (unknown version) on Java 10 (Oracle Corporation) Mac OS X 10.13.6
[apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {"build_date":"2018-11-05T07:58:08Z","build_sha":"dffb98a72a262ca22adad0152f0245ea743ea904","version":"7.0.0-alpha1"}
[main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - service_name: 'elastic-apm-test' (source: Java System Properties)
[main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - log_level: 'DEBUG' (source: Java System Properties)

Make sure to execute some requests to your application before posting your log files. Each request should at least add some lines similar to these in the logs:

[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-2a82cbe3df7a0208f7be6da65be260d1-05e72d045206587a-01 {
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Activating '' 00-2a82cbe3df7a0208f7be6da65be260d1-05e72d045206587a-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.transaction.SpanImpl - startSpan '' 00-2a82cbe3df7a0208f7be6da65be260d1-b2ffa0401105e3d8-01 {
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Activating 'APIRestController#products' 00-2a82cbe3df7a0208f7be6da65be260d1-b2ffa0401105e3d8-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.transaction.SpanImpl - startSpan '' 00-2a82cbe3df7a0208f7be6da65be260d1-49b9d805eca42ec6-01 {
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Activating '' 00-2a82cbe3df7a0208f7be6da65be260d1-49b9d805eca42ec6-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Deactivating 'SELECT' 00-2a82cbe3df7a0208f7be6da65be260d1-49b9d805eca42ec6-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.transaction.SpanImpl - endSpan 'SELECT' 00-2a82cbe3df7a0208f7be6da65be260d1-49b9d805eca42ec6-01
[apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 23)
[apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://127.0.0.1:8200/intake/v2/events
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Deactivating 'APIRestController#products' 00-2a82cbe3df7a0208f7be6da65be260d1-b2ffa0401105e3d8-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.transaction.SpanImpl - endSpan 'APIRestController#products' 00-2a82cbe3df7a0208f7be6da65be260d1-b2ffa0401105e3d8-01
[apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10s
[apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 24)
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Deactivating 'APIRestController#products' 00-2a82cbe3df7a0208f7be6da65be260d1-05e72d045206587a-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - endTransaction 'APIRestController#products' 00-2a82cbe3df7a0208f7be6da65be260d1-05e72d045206587a-01
[apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving TRANSACTION event (sequence 25)

If you don’t see anything in your logs, the technology stack you are using is probably not supported.

After that, you should see logs indicating that the agent has successfully sent data to the APM server:

[apm-request-timeout-timer] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred
[apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving FLUSH event (sequence 26)
[apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Flushing 10912 uncompressed 2667 compressed bytes

If the APM server responds with a 400, it could indicate JSON validation errors. The log would then contain the actual documents which failed to validate:

[apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (±10%)
[apm-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Server returned HTTP response code: 400 for URL: http://127.0.0.1:8200/intake/v2/events
[apm-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {"accepted":13,"errors":[{"message":"Problem validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n  I[#] S[#/allOf/2] allOf failed\n    I[#] S[#/allOf/2/required] missing properties: \"transaction_id\"","document":"{\"span\":{\"name\":\"OpenTracing product span\",\"timestamp\":29352159207,\"id\":\"aeaa7e0ac95acad6\",\"trace_id\":\"d88b5cbfc4536f9a700cd114a53bfeae\",\"parent_id\":\"082fd71ce7e4089a\",\"duration\":17.992,\"context\":{\"tags\":{\"productId\":\"1\"}},\"type\":\"unknown\"}}"}]}

Ideally this should be done outside of production.

Most investigations require using DEBUG log level, thus use DEBUG unless asked for TRACE.

Here, we will refer to the agent log file as /tmp/agent.log, but any other location could be used.

The agent relies on heuristics to define which classes needs to be instrumented or not efficiently to prevent instrumentation overhead.

Those heuristics are based on the packages and class names and are influenced by application_packages and jms_listener_packages configurations. However, if instrumentation is not applied as expected or if you want to investigate creating configurations without proper knowledge of the application internals, it could be relevant to disable those heuristics for investigation:

Sometimes reading the logs is just not enough to debug a problem. As the agent is OpenSource and released on Maven Central, debugging the agent code is really easy.

In order for your IDE to download the sources, first declare a dependency to the agent.

pom.xml.

<dependency>
    <groupId>co.elastic.apm</groupId>
    <artifactId>apm-agent</artifactId>
    <version>${elastic-apm.version}</version>
    <scope>provided</scope>
</dependency>

build.gradle.

compileOnly "co.elastic.apm:apm-agent:$elasticApmVersion"

The most common source of this problem are connection issues between the agent and the APM server.

If the APM server does not receive data from the agent, check if the agent is able to establish a connection to the server. In the agent logs, look out for logs containing Elastic APM server is available and Elastic APM server is not available.

If you see the message Elastic APM server is not available, the agent has problems connecting to the APM server. Check the setting of server_url and make sure the agent is able to connect to the server. Try to execute curl -v <apm-server-url> from the machine the agent is running on. The server should respond with a 200 status code.

If the APM server does not respond successfully, have a look at the APM server logs to verify that the server is actually running. Also make sure to configure your firewalls so that the host the agent runs on can open HTTP connections to the APM server.

By default, transactions are named with the Servlet name that handled the request. Thus, if request does not reach a servlet, the Agent defaults to naming the transaction "unknown route"

There are two reasons why this might happen:

If the proposed fixes do not solve the problem, or if a custom name is required, transaction names can be set manually throughout the request handling flow using our API:

If you are seeing warning like these in your application, it means that you are using a library which has been compiled for a very old version of Java:

org.apache.commons.dbcp.DelegatingStatement uses an unsupported class file version (pre Java 5) and can't be instrumented.
Consider updating to a newer version of that library.

That mostly concerns JDBC drivers. Updating them to a more recent version should resolve the problem.

If you are using a manual setup with a -javaagent flag against an application server and are seeing the Failed to find Premain-Class manifest attribute error and a failure to start, then you might be pointing at the incorrect jar file.

The correct jar file to be pointing at should be in the form of elastic-apm-agent-<version>.jar and further information about how to download this file can be found in the manual setup instructions.

unable to find valid certification path to requested target - server authentication fails. Check out APM Server certificate authentication.

java.net.SocketException: Broken pipe - one option is that client authentication fails. Check out Agent certificate authentication.

With Oracle Weblogic application server wildcard TLS certificates are not allowed by default, when this happens a message error like this is visible in agent logs: Hostname verification failed: HostnameVerifier=weblogic.security.utils.SSLWLSHostnameVerifier. Disabling this extra check can be done with -Dweblogic.security.SSL.ignoreHostnameVerification=true.

For other SSL/TLS related problems, - check out the JSSE troubleshooting section. You can add -Djavax.net.debug=all to the JVM cmd line to get more details about your problem.

More often than not, JVM crashes indicate a JVM bug being surfaced by the installation of the Java agent within the specific configuration of the traced application and it’s dependencies. Therefore, the first thing to try is upgrade the JVM to the latest minor version.

Known issues:

Whenever you encounter a JVM crash, please report through our forum or by opening an issue on our GitHub repository. Look for the crash log (e.g. an hs_err_pid<PID>.log) and provide it when reporting, as well as all factors describing you setup and scenario.

If your JVM gets hang when attaching the Java agent, please create a thread dump (e.g. through jstack) and report through our forum or by opening an issue on our GitHub repository.

If you use jlink to create a custom runtime, make sure the following modules are added: --add-modules java.base,java.logging,java,jdk.zipfs,java.management,jdk.management

In the unlikely event the agent causes disruptions to a production application, you can disable the agent while you troubleshoot.

Using dynamic configuration, you can disable the recording of events by setting recording to false.

If that doesn’t work, you can completely disable the agent by setting enabled to false. You’ll need to restart your application for this change to take effect.