Spans

Transactions can have 0, 1, or many spans. Spans have a transaction.id attribute that refers to their parent transaction. They also have a parent.id attribute that refers to the parent span, or their transaction.

Agents typically automatically instrument a variety of of libraries, but also provide an API for ad hoc instrumentation of specific code paths. A span contains information about a specific code path that has been executed as part of a transaction. This information includes:

As an example, if a database query happens within a sampled transaction, a span describing the database query will be created. The name of this span will contain information about the query itself, and the type of this span will contain information about the database.

Spans are stored in span indices. Note that these indices are separate from transaction indices by default.

For performance reasons, some APM agents can choose to purposefully sample or omit spans. One example of this might be for long running transactions with over 100 spans. These edge cases can overload both the agent and the APM Server. To avoid this, agents will drop spans. When they do this, they notify the server of exactly how many spans were dropped. This note is then passed on to the user in the UI.

Settings affecting dropped spans, and more details on why they might occur, are available in the relevant agent documentation:

Similarly to dropped spans, transactions may have missing spans. This can happen because spans are streamed from the APM Agent to the APM Server separately from their transaction. Unforseen errors may cause spans to go missing. Because the agent notifies the server about how many spans there should be, the number of missing spans is able to be calculated and shown in the UI.