Spans

edit

Spans contain information about the execution of a specific code path. They measure from the start to the end of an activity, and they can have a parent/child relationship with other spans.

Agents automatically instrument a variety of libraries to capture these spans from within your application, but you can also use the Agent API for custom instrumentation of specific code paths.

Among other things, spans can contain:

  • A transaction.id attribute that refers to its parent transaction.
  • A parent.id attribute that refers to its parent span or transaction.
  • Its start time and duration.
  • A name, type, subtype, and action—see the span name/type alignment sheet for span name patterns and examples by APM agent. In addition, some APM agents test against a public span type/subtype spec.
  • An optional stack trace. Stack traces consist of stack frames, which represent a function call on the call stack. They include attributes like function name, file name and path, line number, etc.

Most agents limit keyword fields, like span.id, to 1024 characters, and non-keyword fields, like span.start.us, to 10,000 characters.

Dropped spans
edit

For performance reasons, APM agents can choose to sample or omit spans purposefully. This can be useful in preventing edge cases, like long-running transactions with over 100 spans, that would otherwise overload both the Agent and the APM Server. When this occurs, the APM UI will display the number of spans dropped.

To configure the number of spans recorded per transaction, see the relevant Agent documentation:

Missing spans
edit

Agents stream spans to the APM Server separately from their transactions. Because of this, unforeseen errors may cause spans to go missing. Agents know how many spans a transaction should have; if the number of expected spans does not equal the number of spans received by the APM Server, the APM UI will calculate the difference and display a message.

Data streams
edit

Spans are stored with transactions in the following data streams:

  • Application traces: traces-apm-<namespace>
  • RUM and iOS agent application traces: traces-apm.rum-<namespace>

See Data streams to learn more.

Example span document
edit

This example shows what span documents can look like when indexed in Elasticsearch.

Expand Elasticsearch document
[
    {
        "@timestamp": "2017-05-30T18:53:27.154Z",
        "agent": {
            "name": "elastic-node",
            "version": "3.14.0"
        },
        "ecs": {
            "version": "1.12.0"
        },
        "event": {
            "outcome": "unknown"
        },
        "http": {
            "request": {
                "method": "GET"
            },
            "response": {
                "status_code": 200
            }
        },
        "labels": {
            "span_tag": "something"
        },
        "observer": {
            "hostname": "ix.lan",
            "type": "apm-server",
            "version": "8.0.0"
        },
        "parent": {
            "id": "945254c567a5417e"
        },
        "processor": {
            "event": "span",
            "name": "transaction"
        },
        "service": {
            "environment": "staging",
            "name": "1234_service-12a3"
        },
        "span": {
            "action": "query",
            "db": {
                "instance": "customers",
                "statement": "SELECT * FROM product_types WHERE user_id=?",
                "type": "sql",
                "user": {
                    "name": "readonly_user"
                }
            },
            "duration": {
                "us": 3781
            },
            "http": {
                "method": "GET",
                "response": {
                    "status_code": 200
                }
            },
            "http.url.original": "http://localhost:8000",
            "id": "0aaaaaaaaaaaaaaa",
            "name": "SELECT FROM product_types",
            "stacktrace": [
                {
                    "abs_path": "net.js",
                    "context": {
                        "post": [
                            "    ins.currentTransaction = prev",
                            "    return result",
                            "}"
                        ],
                        "pre": [
                            "  var trans = this.currentTransaction",
                            ""
                        ]
                    },
                    "exclude_from_grouping": false,
                    "filename": "net.js",
                    "function": "onread",
                    "library_frame": true,
                    "line": {
                        "column": 4,
                        "context": "line3",
                        "number": 547
                    },
                    "module": "some module",
                    "vars": {
                        "key": "value"
                    }
                },
                {
                    "exclude_from_grouping": false,
                    "filename": "my2file.js",
                    "line": {
                        "number": 10
                    }
                }
            ],
            "start": {
                "us": 2830
            },
            "subtype": "postgresql",
            "sync": false,
            "type": "db"
        },
        "timestamp": {
            "us": 1496170407154000
        },
        "trace": {
            "id": "945254c567a5417eaaaaaaaaaaaaaaaa"
        },
        "transaction": {
            "id": "945254c567a5417e"
        },
        "url": {
            "original": "http://localhost:8000"
        }
    },
    {
        "@timestamp": "2017-05-30T18:53:42.281Z",
        "agent": {
            "name": "js-base",
            "version": "1.3"
        },
        "destination": {
            "address": "0:0::0:1",
            "ip": "0:0::0:1",
            "port": 5432
        },
        "ecs": {
            "version": "1.12.0"
        },
        "event": {
            "outcome": "unknown"
        },
        "observer": {
            "ephemeral_id": "2f13d8fa-83cd-4356-8123-aabfb47a1808",
            "hostname": "goat",
            "id": "17ad47dd-5671-4c89-979f-ef4533565ba2",
            "type": "apm-server",
            "version": "8.0.0"
        },
        "parent": {
            "id": "85925e55b43f4342"
        },
        "processor": {
            "event": "span",
            "name": "transaction"
        },
        "service": {
            "environment": "staging",
            "name": "serviceabc"
        },
        "span": {
            "action": "query.custom",
            "db": {
                "instance": "customers",
                "statement": "SELECT * FROM product_types WHERE user_id=?",
                "type": "sql",
                "user": {
                    "name": "readonly_user"
                }
            },
            "destination": {
                "service": {
                    "name": "postgresql",
                    "resource": "postgresql",
                    "type": "db"
                }
            },
            "duration": {
                "us": 3781
            },
            "id": "15aaaaaaaaaaaaaa",
            "name": "SELECT FROM product_types",
            "start": {
                "us": 2830
            },
            "subtype": "postgresql",
            "type": "db.postgresql.query"
        },
        "timestamp": {
            "us": 1496170422281000
        },
        "trace": {
            "id": "85925e55b43f4342aaaaaaaaaaaaaaaa"
        },
        "transaction": {
            "id": "85925e55b43f4342"
        }
    },
    {
        "@timestamp": "2017-05-30T18:53:27.154Z",
        "agent": {
            "name": "elastic-node",
            "version": "3.14.0"
        },
        "ecs": {
            "version": "1.12.0"
        },
        "event": {
            "outcome": "unknown"
        },
        "observer": {
            "ephemeral_id": "2f13d8fa-83cd-4356-8123-aabfb47a1808",
            "hostname": "goat",
            "id": "17ad47dd-5671-4c89-979f-ef4533565ba2",
            "type": "apm-server",
            "version": "8.0.0"
        },
        "parent": {
            "id": "945254c567a5417e"
        },
        "processor": {
            "event": "span",
            "name": "transaction"
        },
        "service": {
            "environment": "staging",
            "name": "1234_service-12a3"
        },
        "span": {
            "duration": {
                "us": 32592
            },
            "id": "1aaaaaaaaaaaaaaa",
            "name": "GET /api/types",
            "start": {
                "us": 0
            },
            "subtype": "external",
            "type": "request"
        },
        "timestamp": {
            "us": 1496170407154000
        },
        "trace": {
            "id": "945254c567a5417eaaaaaaaaaaaaaaaa"
        },
        "transaction": {
            "id": "945254c567a5417e"
        }
    },
    {
        "@timestamp": "2017-05-30T18:53:27.154Z",
        "agent": {
            "name": "elastic-node",
            "version": "3.14.0"
        },
        "ecs": {
            "version": "1.12.0"
        },
        "event": {
            "outcome": "unknown"
        },
        "observer": {
            "ephemeral_id": "2f13d8fa-83cd-4356-8123-aabfb47a1808",
            "hostname": "goat",
            "id": "17ad47dd-5671-4c89-979f-ef4533565ba2",
            "type": "apm-server",
            "version": "8.0.0"
        },
        "parent": {
            "id": "945254c567a5417e"
        },
        "processor": {
            "event": "span",
            "name": "transaction"
        },
        "service": {
            "environment": "staging",
            "name": "1234_service-12a3"
        },
        "span": {
            "action": "post",
            "duration": {
                "us": 3564
            },
            "id": "2aaaaaaaaaaaaaaa",
            "name": "GET /api/types",
            "start": {
                "us": 1845
            },
            "subtype": "http",
            "type": "request"
        },
        "timestamp": {
            "us": 1496170407154000
        },
        "trace": {
            "id": "945254c567a5417eaaaaaaaaaaaaaaaa"
        },
        "transaction": {
            "id": "945254c567a5417e"
        }
    },
    {
        "@timestamp": "2017-05-30T18:53:27.154Z",
        "agent": {
            "name": "elastic-node",
            "version": "3.14.0"
        },
        "child": {
            "id": [
                "4aaaaaaaaaaaaaaa"
            ]
        },
        "ecs": {
            "version": "1.12.0"
        },
        "event": {
            "outcome": "unknown"
        },
        "observer": {
            "ephemeral_id": "2f13d8fa-83cd-4356-8123-aabfb47a1808",
            "hostname": "goat",
            "id": "17ad47dd-5671-4c89-979f-ef4533565ba2",
            "type": "apm-server",
            "version": "8.0.0"
        },
        "parent": {
            "id": "945254c567a5417e"
        },
        "processor": {
            "event": "span",
            "name": "transaction"
        },
        "service": {
            "environment": "staging",
            "name": "1234_service-12a3"
        },
        "span": {
            "duration": {
                "us": 13980
            },
            "id": "3aaaaaaaaaaaaaaa",
            "name": "GET /api/types",
            "start": {
                "us": 0
            },
            "type": "request"
        },
        "timestamp": {
            "us": 1496170407154000
        },
        "trace": {
            "id": "945254c567a5417eaaaaaaaaaaaaaaaa"
        },
        "transaction": {
            "id": "945254c567a5417e"
        }
    }
]