Node.js Agent version 3.x

Initial and experimental ECMAScript Module (ESM) support. With the following invocation the APM agent will now be able to instrument modules loaded via import. (See the Node.js introduction to ESM.)

node -r elastic-apm-node/start.js \
  --experimental-loader=elastic-apm-node/loader.mjs \
  server.mjs

# or

NODE_OPTIONS='-r elastic-apm-node/start.js --experimental-loader=elastic-apm-node/loader.mjs'
node server.mjs

The new usage requirement is the --experimental-loader=elastic-apm-node/loader.mjs option. This initial release only includes support for instrumenting a subset of the modules listed at Supported technologies. This set will grow in subsequent versions. Notably, ESM support does not currently work in node v20 — only in recent versions of node v12-v18. ESM support will remain experimental while the Node.js Loaders API is experimental. See ECMAScript module support for full details.

Change the "start.js" export to not start the APM agent inside a Node.js Worker thread.

One way to start the APM agent is via node -r elastic-apm-node/start.js ... or NODE_OPTIONS='-r elastic-apm-node/start.js. When a Node.js Worker thread is started, it inherits the process.execArgv and environment, which results in "start.js" being run in the context of the new thread. Starting an additional APM agent in each new Worker is arguably surprising. For now, "start.js" will avoid starting in a thread. The exact behavior may change in future versions.

One undesirable effect of this change is that explicit use of "start.js" in code (import 'elastic-apm-node/start.js' or require('elastic-apm-node/start.js')) in a Worker will not start the APM agent. Instead, one must use: require('elastic-apm-node').start() or equivalent.

Fix span.context.destination.service.resource for S3 spans to have an "s3/" prefix.

Note: While this is considered a bugfix, but it can potentially be a breaking change in the Kibana APM app: It can break the history of the S3-Spans / metrics for users relying on context.destination.service.resource. If users happen to run agents both with and without this fix (for same or different languages), the same S3-buckets can appear twice in the service map (with and without s3-prefix).

[preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Next.js server-side instrumentation. See the Get started with Next.js document.

This adds instrumentation of the Next.js dev server (next dev) and prod server (next start). The APM transactions for incoming HTTP requests to the server will be named appropriately based on Next.js’s routing — both for user page routes (e.g. GET /a-dynamic-page/[id]) and for internal Next.js routes (e.g. Next.js _next/data route my-page, Next.js Rewrite route /foo -> /bar). As well, exceptions in server-side code (e.g. getServerSideProps, server-side run page handlers, API handlers) will be reported. (#2959)

This is a technical preview to get feedback from Next.js users. The details on how exactly the instrumentation works may change in future versions.

Improve the granularity of data captured about downstream services, e.g. databases, for spans that represent an external call (known as "exit spans"). This data is used for Service Maps and Dependencies in the Kibana APM app.

This is handled via the new span service.target.* fields that replace the deprecated destination.service.resource field (spec). All instrumentations have been updated to set appropriate service target values. If necessary, e.g. for manual instrumentation, a new public span.setServiceTarget(type, name) API has been added to specify these values. (#2882)

The never-public-but-available span.setDestinationContext() has been marked for removal (using it will process.emitWarning()). Users of this internal method should switch to the public span.setServiceTarget(type, name).

As part of this change, improvements have been made to some module instrumentations:

Improve the captured information for Elasticsearch client instrumentation. For all outgoing Elasticsearch client requests, the full HTTP url is now captured (stored in the "url.original" field). For Elasticsearch requests that do a search, the outgoing request body is captured (to the "span.db.statement" field) as before, but the format has changed to only hold the request body. Before this change the "span.db.statement" would also hold any HTTP query parameters. These are now more naturally captured in "url.original". (#2019)

This change also introduces the elasticsearchCaptureBodyUrls configuration option to enable controlling which Elasticsearch REST API paths are considered for request body capture. (#2873)

The agent will now use AsyncLocalStorage for run-context tracking in new enough versions of Node.js (versions >=14.5 and >=12.19). This can reduce overhead from using the APM agent, especially in Promise-heavy applications. (#2786)

This also adds a new contextManager configuration option to control which mechanism the agent uses for run-context tracking. It replaces the, now deprecated, asyncHooks configuration option. If you experience problems with the new AsyncLocalStorage-based tracking, you can restore the older behavior with contextManager: "asynchooks".

Add Span Links support. (#2673)

The transaction.startSpan() and apm.startTransaction() public APIs now accept a links option for specify links. The OpenTelemetry Bridge also supports specifying links during span creation (with the limitation that span link attributes are not supported).

Add a parent option to agent.captureError(err[, options][, cb]) to allow passing in a Transaction or Span to use as the parent for the error. Before this change the current span or transaction, if any, was always used.

This option is not documented in the user docs, nor added to the TypeScript types, because it is only expected to be useful for coming OTel Bridge work.

Add support for span compression. (#2100, #2604)

By default, consecutive (sibling) exit spans of the same name, type, subtype, and destination with a duration of less than 50ms will be compressed into a single composite span. A possible case is the N+1 query problem. Traces with many consecutive matching spans will be represented — both in data and the APM UI — more efficiently.

Span compression can be disabled or matching behavior configured with the spanCompression* configuration options.

Added a new config option spanStackTraceMinDuration that replaces both captureSpanStackTraces and spanFramesMinDuration. The latter two are now deprecated, but still supported. If spanStackTraceMinDuration is specified, then any value for the deprecated two options will be ignored.

There is a significant change in default behavior of the APM agent. If none of these configuration options is specified, then the default (spanStackTraceMinDuration: -1) is that stack traces are not collected and reported for any spans. This change in default behavior was made because the CPU performance impact of collecting span stack traces was found to be too high in practice for busy and/or complex applications. This is mentioned in the "Breaking changes" section to highlight the change, but it is not considered breaking in general. The impact is that the "Stack Trace" tab in the "Span details" view in the Kibana APM app will be empty. This was already the case for some spans based on span duration. (#2565)

Implement the explicit signaling of Lambda invocation completion to the Elastic AWS Lambda Extension. This improves data flushing in a Lambda environment to ensure tracing data is only sent when the Lambda is active. This avoids possible tracing data loss while a Lambda VM is frozen. (#2485)

However, because this change triggers a bug in the extension, this version of the APM Node.js Agent must only be used with versions of the AWS Lambda Extension after v0.0.3.

Elasticsearch spans (from elasticsearch, @elastic/elasticsearch, and @elastic/elasticsearch-canary instrumentation) will no longer have an HTTP child span(s) for the underlying HTTP request. This is listed in this section to provide awareness in case some users have custom analysis of APM trace data that expects those HTTP spans.

Per the APM Agent spec for exit spans, Elasticsearch spans are now marked as exit spans and as a result, HTTP child spans are suppressed. (#2000)

As part of this change, some HTTP context has been added to Elasticsearch spans, when available: the HTTP response status_code, and the size of the response body (encoded_body_size). (#2484)

Fix an issue where the agent’s async task tracking could cause the user’s application to use too much memory. In cases where an application is under sustained load and is running in a memory constrained container, this looked like a memory leak.

This high memory usage could happen when application code starts async tasks (e.g. Promises, setTimeouts, async I/O) that outlive the APM Transaction (typically an HTTP request handler). The agent’s async task tracking keeps a reference to the APM Transaction (and any APM Spans) until the async task ends, thus extending the lifetime of those APM objects and the references they hold — in particular, HTTP request and response objects. This could lead to higher memory usage.

With this change, those references are removed when APM Transactions and Spans are ended, and agent memory usage is now the same as what it was before v3.24.0 when this issue was introduced. (#2528, #2489)

Fixes for ioredis instrumentation (#2460):

Fixes for redis instrumentation:

Change disableSend to no longer skip internal processing work. It now only disables communication with APM Server. Use contextPropagationOnly if your use case is to limit the APM agent’s processing to the minimum to support context propagation and log correlation.

This is listed under "Breaking changes" as a heads-up. The only possible negative result of this disableSend change is some extra CPU processing time by the agent. There is no outward functionality change.

Add contextPropagationOnly configuration option. This supports the use case of using the APM agent to propagate HTTP trace-context and to support log-correlation (adding trace.id et al fields to log records) without an APM server, and to otherwise reduce the processing time of the agent. (#2393)

This is similar to disableSend, but differs in that contextPropagationOnly: true tells the agent to skip unnecessary work.

Zero configuration support. The only required agent configuration option is serviceName. Normally the agent will attempt to infer serviceName for the "name" field in a package.json file. However, that could fail. With this version, the agent will cope with: a scoped package name (@scope/name is normalized to scope-name), a "name" that isn’t a valid serviceName, not being able to find a "package.json" file, etc. Ultimately it will fallback to "nodejs_service". (#1944)

One consequence of this change is that apm.getServiceName() will return undefined until the agent is started (check with apm.isStarted()).

A significant change was made to internal run context tracking (a.k.a. async context tracking). There are no configuration changes or API changes for custom instrumentation. (#2181)

One behavior change is that multiple spans created synchronously (in the same async task) will form parent/child relationships; before this change they would all be siblings. This fixes HTTP child spans of Elasticsearch and aws-sdk automatic spans to properly be children. (#1889)

Another behavior change is that a span B started after having ended span A in the same async task will no longer be a child of span A. (#1964)

This fixes an issue with context binding of EventEmitters, where removeListener would fail to actually remove if the same handler function was added to multiple events.

Guard against a negative value of metricsInterval, which can lead to high CPU usage as metrics are collected as fast as possible. Also ensure no metrics collection can happen if metricsInterval="0s" as intended. Before this change it was possible for some metric collection to still happen, even though none would be reported. (#2330)

This change also guards against negative and invalid values in the following configuration options: abortedErrorThreshold, apiRequestTime, and serverTimeout. If an invalid value is given, then will fallback to their default value.

Add the longFieldMaxLength integer configuration option (default 10000). Specific transaction/span/error fields (see the list below) will be truncated at this number of unicode characters. (#2193, #1921)

The errorMessageMaxLength configuration option is now deprecated, but still supported. Users should switch to using longFieldMaxLength. If errorMessageMaxLength is not specified, truncation of error messages will now use the longFieldMaxLength value.

Note that ultimately the maximum length of any tracing field is limited by the max_event_size configured for the receiving APM server.

The fields affected by longFieldMaxLength are:

Performance improvements have been made in error and stacktrace capture (#2094). This also included in two bug fixes:

The handling of sending events (transactions, spans, errors) to APM server has improved in a few ways. During temporary spikes in load and/or an APM server that is unresponsive, the agent will buffer a number of events and drop them above a certain limit (configurable via maxQueueSize). This helps ensure the agent does not overly consume memory and CPU. As well, the agent will now backoff when the APM server errors. Finally, improved error handling means it will terminate failing requests to the APM server more quickly.

Note: v1 of this agent (EOL’d 2 years ago), included a maxQueueSize config variable with a different meaning. If you have a lingering usage of that setting (also possibly as the ELASTIC_APM_MAX_QUEUE_SIZE environment variable), then it should be removed.

The APM agent’s own internal logging now uses structured JSON logging using the pino API, and formatted in ecs-logging format. The log records on stdout are now single-line JSON objects. For example:

# Before
APM Server transport error (ECONNREFUSED): connect ECONNREFUSED 127.0.0.1:8200

# After
{"log.level":"error","@timestamp":"2021-03-19T00:21:17.571Z","log":{"logger":"elastic-apm-node"},
"ecs":{"version":"1.6.0"},"message":"APM Server transport error (ECONNREFUSED): connect ECONNREFUSED 127.0.0.1:8200"}

Pretty formatting (and filtering) on the console may be done via the ecslog tool.

A custom logger is still supported as before. However, a non-pino custom logger will only receive the "message" field, and not structured log fields as they are added over time.

feat: Add automatic instrumentation of the @elastic/elasticsearch package #1870

The instrumentation of the legacy "elasticsearch" package has also changed slightly to commonalize:

feat: Add sanitize_field_names configuration option.
Allows users to configure a list of wildcard patterns to remove items from the agent’s HTTP header and application/x-www-form-urlencoded payloads. #1898

fix: Fix parsing of comma-separated strings for relevant config vars to allow whitespace around the commas. E.g.:

export ELASTIC_APM_TRANSACTION_IGNORE_URLS='/ping, /metrics*'

Config vars affected are: disableInstrumentations, transactionIgnoreUrls addPatch, and globalLabels.