« Google Cloud Storage Input HTTP JSON input »

› › ›

HTTP Endpoint input

The HTTP Endpoint input initializes a listening HTTP server that collects incoming HTTP POST requests containing a JSON body. The body must be either an object or an array of objects, otherwise a Common Expression Language expression that converts the the JSON body to these types can be provided. Any other data types will result in an HTTP 400 (Bad Request) response. For arrays, one document is created for each object in the array.

gzip encoded request bodies are supported if a Content-Encoding: gzip header is sent with the request.

This input can for example be used to receive incoming webhooks from a third-party application or service.

Multiple endpoints may be assigned to a single address and port, and the HTTP Endpoint input will resolve requests based on the URL pattern configuration. If multiple endpoints are configured on a single address they must all have the same TLS configuration, either all disabled or all enabled with identical configurations.

These are the possible response codes from the server.

HTTP Response Code	Name	Reason
200	OK	Returned on success.
400	Bad Request	Returned if JSON body decoding fails or if `wait_for_completion_timeout` query validation fails.
401	Unauthorized	Returned when basic auth, secret header, or HMAC validation fails.
405	Method Not Allowed	Returned if methods other than POST are used.
406	Not Acceptable	Returned if the POST request does not contain a body.
415	Unsupported Media Type	Returned if the Content-Type is not application/json. Or if Content-Encoding is present and is not gzip.
500	Internal Server Error	Returned if an I/O error occurs reading the request.
503	Service Unavailable	Returned if the length of the request body would take the total number of in-flight bytes above the configured `max_in_flight_bytes` value.
504	Gateway Timeout	Returned if a request publication cannot be ACKed within the required timeout.

The endpoint will enforce end-to-end ACK when a URL query parameter wait_for_completion_timeout with a duration is provided. For example http://localhost:8080/?wait_for_completion_timeout=1m will wait up to 1 minute for the event to be published to the cluster and then return the user-defined response message. In the case that the publication does not complete within the timeout duration, the HTTP response will have a 504 Gateway Timeout status code. The syntax for durations is a number followed by units which may be h, m and s. No other HTTP query is accepted. If another query parameter is provided or duration syntax is incorrect, the request will fail with an HTTP 400 "Bad Request" status.

Example configurations:

Basic example:

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080

Custom response example:

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080
  response_code: 200
  response_body: '{"message": "success"}'
  url: "/"
  prefix: "json"

Map request to root of document example:

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080
  prefix: "."

Multiple endpoints example:

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080
  url: "/open/"
  tags: [open]
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080
  url: "/admin/"
  basic_auth: true
  username: adminuser
  password: somepassword
  tags: [admin]

Disable Content-Type checks

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  content_type: ""
  prefix: "json"

Basic auth and SSL example:

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080
  ssl.enabled: true
  ssl.certificate: "/home/user/server.pem"
  ssl.key: "/home/user/server.key"
  ssl.verification_mode: "none"
  ssl.certificate_authority: "/home/user/ca.pem"
  basic_auth: true
  username: someuser
  password: somepassword

Authentication or checking that a specific header includes a specific value

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080
  secret.header: someheadername
  secret.value: secretheadertoken

Validate webhook endpoint for a specific provider using CRC

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080
  secret.header: someheadername
  secret.value: secretheadertoken
  crc.provider: webhookProvider
  crc.secret: secretToken

Validate a HMAC signature from a specific header

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080
  hmac.header: "X-Hub-Signature-256"
  hmac.key: "password123"
  hmac.type: "sha256"
  hmac.prefix: "sha256="

Preserving original event and including headers in document

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080
  preserve_original_event: true
  include_headers: ["TestHeader"]

Common Expression Language example:

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: 192.168.1.1
  listen_port: 8080
  program: |
    obj.records.map(r, {
     "requestId": obj.requestId,
     "timestamp": string(obj.timestamp),
     "event": r,
    })

This example would allow handling of a JSON body that is an object containing more than one event that each should be ingested as separate documents with the common timestamp and request ID:

{
  "requestId": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f",
  "timestamp": 1578090901599,
  "records": [
    {
      "data": "event record 1"
    },
    {
      "data": "event record 2"
    }
  ]
}

Metric	Description
`bind_address`	Bind address of input.
`route`	HTTP request route of the input.
`is_tls_connection`	Whether the input is listening on a TLS connection.
`api_errors_total`	Number of API errors.
`batches_received_total`	Number of event arrays received.
`batches_published_total`	Number of event arrays published.
`batches_acked_total`	Number of event arrays ACKed.
`events_published_total`	Number of events published.
`size`	Histogram of request content lengths.
`batch_size`	Histogram of the received event array length.
`batch_processing_time`	Histogram of the elapsed successful batch processing times in nanoseconds (time of receipt to time of ACK for non-empty batches).
`batch_ack_time`	Histogram of the elapsed successful batch ACKing times in nanoseconds (time of handler start to time of ACK for non-empty batches).

HTTP Endpoint input

HTTP Endpoint input

Configuration options

basic_auth

username

password

secret.header

secret.value

hmac.header

hmac.key

hmac.type

hmac.prefix

content_type

max_in_flight_bytes

retry_after

program

response_code

response_body

listen_address

listen_port

url

prefix

include_headers

preserve_original_event

crc.provider

crc.secret

method

tracer.enabled

tracer.filename

tracer.maxsize

tracer.maxage

tracer.maxbackups

tracer.localtime

tracer.compress

Metrics

Common options

enabled

tags

fields

fields_under_root

processors

pipeline

keep_null

index

publisher_pipeline.disable_host

`basic_auth`

`username`

`password`

`secret.header`

`secret.value`

`hmac.header`

`hmac.key`

`hmac.type`

`hmac.prefix`

`content_type`

`max_in_flight_bytes`

`retry_after`

`program`

`response_code`

`response_body`

`listen_address`

`listen_port`

`url`

`prefix`

`include_headers`

`preserve_original_event`

`crc.provider`

`crc.secret`

`method`

`tracer.enabled`

`tracer.filename`

`tracer.maxsize`

`tracer.maxage`

`tracer.maxbackups`

`tracer.localtime`

`tracer.compress`

`enabled`

`tags`

`fields`

`fields_under_root`

`processors`

`pipeline`

`keep_null`

`index`

`publisher_pipeline.disable_host`