Configure lightweight monitors

edit

Configure lightweight monitors

edit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

In the Synthetics app you can monitor the status of network endpoints using the following lightweight checks:

  • HTTP: Monitor your website. The HTTP monitor checks to make sure specific endpoints return the correct status code and display the correct text.
  • ICMP: Check the availability of your hosts. The ICMP monitor uses ICMP (v4 and v6) Echo Requests to check the network reachability of the hosts you are pinging. This will tell you whether the host is available and connected to the network, but doesn’t tell you if a service on the host is running or not.
  • TCP: Monitor the services running on your hosts. The TCP monitor checks individual ports to make sure the service is accessible and running.

Lightweight monitors can be configured using either the Synthetics app or Project monitors.

Synthetics app

edit

To use the UI in the Synthetics app, go to the Synthetics app in Kibana to create and configure monitors. For step-by-step instructions, refer to Use the Synthetics app.

Synthetics Create monitor UI

Project monitors

edit

To use YAML files in to create project monitors, set up a project and configure monitors in YAML files in the lightweight directory.

In each YAML file, define a set of monitors to check your remote hosts. Each monitor item is an entry in a YAML list and begins with a dash (-). You can define the type of monitor to use, the hosts to check, and other optional settings.

The following example configures three monitors checking via the http, icmp, and tcp protocols and demonstrates how to use TCP Echo response verification:

heartbeat.monitors:
- type: http
  name: Todos Lightweight
  id: todos-lightweight
  urls: "https://elastic.github.io/synthetics-demo/"
  schedule: '@every 1m'
- type: icmp
  id: ping-myhost
  name: My Host Ping
  hosts: ["myhost"]
  schedule: '@every 5m'
- type: tcp
  id: myhost-tcp-echo
  name: My Host TCP Echo
  hosts: ["myhost:777"]  # default TCP Echo Protocol
  check.send: "Check"
  check.receive: "Check"
  schedule: '@every 60s'

There are some common monitor configuration options that are the same for all lightweight monitor types. For a complete list, refer to Common options.

Each monitor type also has additional configuration options that are specific to that type. Refer to:

The tcp and http monitor types both support SSL/TLS and some proxy settings.

Common options

edit

You can specify the following options when defining a synthetic monitor in any location. These options are the same for all monitors. Each monitor type has additional configuration options that are specific to that monitor type.

Option (type) Description

type
("http" | "icmp" | "tcp")

Required. The type of monitor to run. One of:

  • http: Connects via HTTP and optionally verifies that the host returns the expected response.
  • icmp: Uses an ICMP (v4 and v6) Echo Request to ping the configured hosts. Requires special permissions or root access.
  • tcp: Connects via TCP and optionally verifies the endpoint by sending and/or receiving a custom payload.

id
(string)

Required. A unique identifier for this configuration. This should not change with edits to the monitor configuration regardless of changes to any config fields.

Examples:

id: uploader-service
id: http://example.net

When querying against indexed monitor data this is the field you will be aggregating with. It appears in the exported fields as monitor.id.

If you do not set an id explicitly, the monitor’s config will be hashed and a generated value will be used. This value will change with any options change to this monitor making aggregations over time between changes impossible. For this reason, it’s recommended that you set this manually.

name
(string)

Human readable name for this monitor.

Examples:

name: Uploader service
name: Example website

service.name
(string)

APM service name for this monitor. Corresponds to the service.name ECS field. Set this when monitoring an app that is also using APM to enable integrations between Synthetics and APM data in Kibana.

enabled
(boolean)

Whether the monitor is enabled.

Default: true

Example:

enabled: false

schedule
(duration)

Required. The task schedule.

Schedules with less than 1 minute resolution will be saved to the nearest minute. For example, @every 5s will be changed to @every 60s when the monitor is pushed using the CLI.

Example:
Run the task every 5 minutes from the time the monitor was started.

schedule: @every 5m

timeout
(duration)

The total running time for each ping test. This is the total time allowed for testing the connection and exchanging data.

Default: 16s

Example:

timeout: 2m

tags
(list of strings)

A list of tags that will be sent with the monitor event.

Examples:

tags:
  - tag one
  - tag two
tags: ["tag one", "tag two"]

HTTP options

edit

The options described here configure Synthetics to connect via HTTP and optionally verify that the host returns the expected response.

Valid options for HTTP monitors include all common options and the following HTTP-specific options:

Option (type) Description

hosts
(list of strings)

Required. A list of URLs to ping.

max_redirects
(number)

The total number of redirections Synthetics will follow.

By default, Synthetics will not follow redirects, but will report the status of the redirect. If set to a number greater than 0, Synthetics will follow that number of redirects.

When this option is set to a value greater than 0, the monitor.ip field will no longer be reported, as multiple DNS requests across multiple IPs may return multiple IPs. Fine-grained network timing data will also not be recorded, as with redirects that data will span multiple requests. Specifically the fields http.rtt.content.us, http.rtt.response_header.us, http.rtt.total.us, http.rtt.validate.us, http.rtt.write_request.us and dns.rtt.us will be omitted.

Default: 0

proxy_url
(string)

The HTTP proxy URL. This setting is optional.

Example:

http://proxy.mydomain.com:3128

username
(string)

The username for authenticating with the server. The credentials are passed with the request. This setting is optional.

You need to specify credentials when your check.response settings require it. For example, you can check for a 403 response (check.response.status: [403]) without setting credentials.

password
(string)

The password for authenticating with the server. This setting is optional.

ssl
(SSL)

The TLS/SSL connection settings for use with the HTTPS endpoint. If you don’t specify settings, the system defaults are used.

Example:

- type: http
  id: my-http-service
  name: My HTTP Service
  hosts: ["https://myhost:443"]
  schedule: '@every 5s'
  ssl:
    certificate_authorities: ['/etc/ca.crt']
    supported_protocols: ["TLSv1.0", "TLSv1.1", "TLSv1.2"]

headers
(boolean)

Controls the indexing of the HTTP response headers http.response.body.headers field. Set response.include_headers to false to disable.

Default: true

response

Controls the indexing of the HTTP response body contents to the http.response.body.contents field.

include_body ("on_error" | "never" | "always")

Set response.include_body to one of the options listed below.

  • on_error: Include the body if an error is encountered during the check. This is the default.
  • never: Never include the body.
  • always: Always include the body with checks.

check

request

An optional request to send to the remote host. Under check.request, specify these options:

method ("HEAD" | "GET" | "POST" | "OPTIONS")
The HTTP method to use.
headers (HTTP headers)
A dictionary of additional HTTP headers to send. By default Synthetics will set the User-Agent header to identify itself.
body (string)
Optional request body content.

Example: This monitor POSTs an x-www-form-urlencoded string to the endpoint /demo/add.

check.request:
  method: POST
  headers:
    'Content-Type': 'application/x-www-form-urlencoded'
  # urlencode the body:
  body: "name=first&email=someemail%40someemailprovider.com"
response

The expected response.

Under check.response, specify these options:

status (list of strings)

A list of expected status codes. 4xx and 5xx codes are considered down by default. Other codes are considered up.

Example:

check.response:
  status: [200, 201]
headers (HTTP headers)
The required response headers.
body.positive (list of strings)

A list of regular expressions to match the body output. Only a single expression needs to match.

Example:

This monitor examines the response body for the strings foo or Foo:

check.response:
  status: [200, 201]
  body:
    positive:
      - foo
      - Foo
body.negative (list of strings)

A list of regular expressions to match the body output negatively. Return match failed if single expression matches. HTTP response bodies of up to 100MiB are supported.

This monitor examines match successfully if there is no bar or Bar at all, examines match failed if there is bar or Bar in the response body:

Example:

check.response:
  status: [200, 201]
  body:
    negative:
      - bar
      - Bar

Example:

This monitor examines match successfully only when foo or Foo in body AND no bar or Bar in body:

check.response:
  status: [200, 201]
  body:
    positive:
      - foo
      - Foo
    negative:
      - bar
      - Bar

ICMP options

edit

The options described here configure Synthetics to use ICMP (v4 and v6) Echo Requests to check the configured hosts. On most platforms you must execute Synthetics with elevated permissions to perform ICMP pings.

On Linux, regular users may perform pings if the right file capabilities are set. Run sudo setcap cap_net_raw+eip /path/to/heartbeat to grant Synthetics ping capabilities on Linux. Alternatively, you can grant ping permissions to the user being used to run Synthetics. To grant ping permissions in this way, run sudo sysctl -w net.ipv4.ping_group_range='myuserid myuserid'.

Other platforms may require Synthetics to run as root or administrator to execute pings.

Valid options for ICMP monitors include all common options and the following ICMP-specific options:

Option (type) Description

hosts
(list of strings)

Required. A list of hosts to ping.

Example:

hosts: ["myhost"]

wait
(duration)

The duration to wait before emitting another ICMP Echo Request if no response is received.

Default: 1s

Example:

wait: 1m

TCP options

edit

The options described here configure Synthetics to connect via TCP and optionally verify the endpoint by sending and/or receiving a custom payload.

Valid options for TCP monitors include all common options and the following TCP-specific options:

Option (type) Description

hosts
(list of strings)

Required. A list of hosts to ping. The entries in the list can be:

  • A plain host name, such as localhost, or an IP address.
    If you specify this option, you must also specify a value for ports. If the monitor is configured to use SSL, Synthetics establishes an SSL/TLS-based connection. Otherwise, it establishes a plain TCP connection.
  • A hostname and port, such as localhost:12345.
    Synthetics connects to the port on the specified host. If the monitor is configured to use SSL, Synthetics establishes an SSL/TLS-based connection. Otherwise, it establishes a TCP connection.
  • A full URL using the syntax scheme://<host>:[port], where:

    • scheme is one of tcp, plain, ssl or tls. If tcp or plain is specified, Synthetics establishes a TCP connection even if the monitor is configured to use SSL. If tls or ssl is specified, Synthetics establishes an SSL connection. However, if the monitor is not configured to use SSL, the system defaults are used (currently not supported on Windows).
    • host is the hostname.
    • port is the port number. If port is missing in the URL, the ports setting is required.

Examples:

hosts: ["localhost"]
hosts: ["localhost:8000"]
hosts: ["tcp://localhost:8000"]

ports
(list of strings)

A list of ports to ping if the host specified in hosts does not contain a port number. It is generally preferable to use a single value here, since each port will be monitored using a separate id, with the given id value, used as a prefix in the Synthetics data, and the configured name shared across events sent via this check.

Example:

hosts: ["localhost"]
ports: [80, 9200, 5044]

check

An optional payload string to send to the remote host and the expected answer. If no payload is specified, the endpoint is assumed to be available if the connection attempt was successful. If send is specified without receive, any response is accepted as OK. If receive is specified without send, no payload is sent, but the client expects to receive a payload in the form of a "hello message" or "banner" on connect.

Example:

check.send: 'Hello World'
check.receive: 'Hello World'
check:
  send: 'Hello World'
  receive: 'Hello World'

proxy_url

The URL of the SOCKS5 proxy to use when connecting to the server. The value must be a URL with a scheme of socks5://.

If the SOCKS5 proxy server requires client authentication, then a username and password can be embedded in the URL.

When using a proxy, hostnames are resolved on the proxy server instead of on the client. You can change this behavior by setting the proxy_use_local_resolver option.

Examples:

A proxy URL that requires client authentication:

proxy_url: socks5://user:password@socks5-proxy:2233

proxy_use_local_resolver
(boolean)

A Boolean value that determines whether hostnames are resolved locally instead of being resolved on the proxy server. The default value is false, which means that name resolution occurs on the proxy server.

Default: false

ssl
(SSL)

The TLS/SSL connection settings. If the monitor is configured to use SSL, it will attempt an SSL handshake. If check is not configured, the monitor will only check to see if it can establish an SSL/TLS connection. This check can fail either at TCP level or during certificate validation.

Example:

ssl:
  certificate_authorities: ['/etc/ca.crt']
  supported_protocols: ["TLSv1.0", "TLSv1.1", "TLSv1.2"]

Also see Configure SSL for a full description of the ssl options.

Data types reference

edit

Values of configuration settings are interpreted as required by Synthetics. If a value can’t be correctly interpreted as the required type - for example a string is given when a number is required - Synthetics will fail to start up.

Boolean
edit

Boolean values can be either true or false. Alternative names for true are yes and on. Instead of false the values no and off can be used.

enabled: true
disabled: false
Number
edit

Number values require you to enter the number without single or double quotes.

integer: 123
negative: -1
float: 5.4

Some settings only support a restricted number range.

String
edit

In YAML, multiple styles of string definitions are supported: double-quoted, single-quoted, unquoted.

The double-quoted style is specified by surrounding the string with ". This style provides support for escaping unprintable characters using \, but comes at the cost of having to escape \ and " characters.

The single-quoted style is specified by surrounding the string with '. This style supports no escaping (use '' to quote a single quote). Only printable characters can be used when using this form.

Unquoted style requires no quotes, but does not support any escaping and can’t include any symbol that has a special meaning in YAML.

Single-quoted style is recommended when defining regular expressions, event format strings, windows file paths, or non-alphabetical symbolic characters.

Duration
edit

Durations require a numeric value with optional fraction and required unit. Valid time units are ns, us, ms, s, m, h. Sometimes features based on durations can be disabled by using zero or negative durations.

duration1: 2.5s
duration2: 6h
duration_disabled: -1s
Regular expression
edit

Regular expressions are special strings that are compiled into regular expressions at load time.

As regular expressions and YAML use \ for escaping characters in strings, it’s highly recommended to use single quoted strings when defining regular expressions. When single quoted strings are used, the \ character is not interpreted by YAML parser as an escape symbol.