Elastic APM events intake API

edit

Most users do not need to interact directly with the events intake API.

The events intake API is what we call the internal protocol that APM agents use to talk to the APM Server. Agents communicate with the Server by sending events — captured pieces of information — in an HTTP request. Events can be:

  • Transactions
  • Spans
  • Errors
  • Metrics

Each event is sent as its own line in the HTTP request body. This is known as newline delimited JSON (NDJSON).

With NDJSON, agents can open an HTTP POST request and use chunked encoding to stream events to the APM Server as soon as they are recorded in the agent. This makes it simple for agents to serialize each event to a stream of newline delimited JSON. The APM Server also treats the HTTP body as a compressed stream and thus reads and handles each event independently.

See the APM data model to learn more about the different types of events.

Endpoints

edit

APM Server exposes the following endpoints for Elastic APM agent data intake:

Name Endpoint

APM agent event intake

/intake/v2/events

RUM event intake (v2)

/intake/v2/rum/events

RUM event intake (v3)

/intake/v3/rum/events

Request

edit

Send an HTTP POST request to the APM Server intake/v2/events endpoint:

http(s)://{hostname}:{port}/intake/v2/events

From version 8.5.0 onwards, the APM Server supports asynchronous processing of batches. To request asynchronous processing the async query parameter can be set in the POST request to the intake/v2/events endpoint:

http(s)://{hostname}:{port}/intake/v2/events?async=true

Since asynchronous processing defers some of the event processing to the background and takes place after the client has closed the request, some errors can’t be communicated back to the client and are logged by the APM Server. Furthermore, asynchronous processing requests will only be scheduled if the APM Server can service the incoming request, requests that cannot be serviced will receive an internal error 503 "queue is full" error.

For RUM send an HTTP POST request to the APM Server intake/v3/rum/events endpoint instead:

http(s)://{hostname}:{port}/intake/v3/rum/events

Response

edit

On success, the server will respond with a 202 Accepted status code and no body.

Keep in mind that events can succeed and fail independently of each other. Only if all events succeed does the server respond with a 202.

Errors

edit

There are two types of errors that the APM Server may return to an agent:

  • Event related errors (typically validation errors)
  • Non-event related errors

The APM Server processes events one after the other. If an error is encountered while processing an event, the error encountered as well as the document causing the error are added to an internal array. The APM Server will only save 5 event related errors. If it encounters more than 5 event related errors, the additional errors will not be returned to agent. Once all events have been processed, the error response is sent.

Some errors, not relating to specific events, may terminate the request immediately. For example: IP rate limit reached, wrong metadata, etc. If at any point one of these errors is encountered, it is added to the internal array and immediately returned.

An example error response might look something like this:

{
  "errors": [
    {
      "message": "<json-schema-err>", 
      "document": "<ndjson-obj>" 
    },{
      "message": "<json-schema-err>",
      "document": "<ndjson-obj>"
    },{
      "message": "<json-decoding-err>",
      "document": "<ndjson-obj>"
    },{
      "message": "too many requests" 
    },
  ],
  "accepted": 2320 
}

An event related error

The document causing the error

An immediately returning non-event related error

The number of accepted events

If you’re developing an agent, these errors can be useful for debugging.

Event API Schemas

edit

The APM Server uses a collection of JSON Schemas for validating requests to the intake API: