Topbeat was replaced by Metricbeat in 5.0. To learn more about Metricbeat, see the Metricbeat documentation.
« Input Shipper »
Elastic Docs ›Topbeat Reference [1.0.1] ›Configuration Options

Output

edit

Output

edit

You can configure multiple outputs for exporting the correlated transactions. Currently the following output types are supported:

You can enable one or multiple outputs at a time. The output plugins are responsible for sending the transaction data in JSON format to the next step in the pipeline. They are also responsible for maintaining the network topology.

Elasticsearch Output

edit

When you specify Elasticsearch for the output, the Beat sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API.

Example configuration:

output:
  elasticsearch:
    # The Elasticsearch cluster
    hosts: ["http://localhost:9200"]

    # Comment this option if you don't want to store the topology in
    # Elasticsearch. The default is false.
    # This option makes sense only for Packetbeat
    # save_topology: false

    # Optional index name. The default is "topbeat" and generates
    # [topbeat-]YYYY.MM.DD keys.
    index: "topbeat"

    # tls configuration. By default is off.
    tls:
      # List of root certificates for HTTPS server verifications
      certificate_authorities: ["/etc/pki/root/ca.pem"]

      # Certificate for TLS client authentication
      certificate: "/etc/pki/client/cert.pem"

      # Client Certificate Key
      certificatekey: "/etc/pki/client/cert.key"

To enable SSL, just add https to all URLs defined under hosts.

output:
  elasticsearch:
    # The Elasticsearch cluster
    hosts: ["https://localhost:9200"]

    # Comment this option if you don't want to store the topology in
    # Elasticsearch. The default is false.
    # This option makes sense only for Packetbeat
    # save_topology: false

    # HTTP basic auth
    username: "admin"
    password: "s3cr3t"

If the Elasticsearch nodes are defined by IP:PORT, then add protocol: https to the yaml file.

output:
  elasticsearch:
    # The Elasticsearch cluster
    hosts: ["localhost"]

    # Optional http or https. Default is http
    protocol: "https"

    # Comment this option if you don't want to store the topology in
    # Elasticsearch. The default is false.
    # This option makes sense only for Packetbeat
    # save_topology: false

    # HTTP basic auth
    username: "admin"
    password: "s3cr3t"

hosts

edit

The list of Elasticsearch nodes to connect to. The events are distributed to these nodes in round robin order. If one node becomes unreachable, the event is automatically sent to another node. Each Elasticsearch node can be defined as a URL or IP:PORT. For example: http://192.15.3.2, https://es.found.io:9230 or 192.24.3.2:9300. If no port is specified, 9200 is used.

When a node is defined as an IP:PORT, the scheme and path are taken from the protocol and path config options.

output:
  elasticsearch:
    # The Elasticsearch cluster
    hosts: ["10.45.3.2:9220", "10.45.3.1:9230"]

    # Optional http or https. Default is http
    protocol: https

    # HTTP Path at which each Elasticsearch server lives
    path: /elasticsearch

In the previous example, the Elasticsearch nodes are available at https://10.45.3.2:9220/elasticsearch and https://10.45.3.1:9230/elasticsearch.

worker

edit

The number of workers per configured host publishing events to Elasticsearch. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).

host (DEPRECATED)

edit

The host of the Elasticsearch server. This option is deprecated because it is replaced by hosts.

port (DEPRECATED)

edit

The port of the Elasticsearch server. This option is deprecated because it is replaced by hosts.

username

edit

The basic authentication username for connecting to Elasticsearch.

password

edit

The basic authentication password for connecting to Elasticsearch.

protocol

edit

The name of the protocol Elasticsearch is reachable on. The options are: http or https. The default is http. However, if you specify a URL for hosts, the value of protocol is overridden by whatever scheme you specify in the URL.

path

edit

An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where Elasticsearch listens behind an HTTP reverse proxy that exports the API under a custom prefix.

index

edit

The index root name to write events to. The default is the Beat name. For example "topbeat" generates "[topbeat-]YYYY.MM.DD" indexes (for example, "topbeat-2015.04.26").

max_retries

edit

The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. Some Beats, such as Filebeat, ignore the max_retries setting and retry until all events are published.

Set max_retries to a value less than 0 to retry until all events are published.

The default is 3.

bulk_max_size

edit

The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50.

timeout

edit

The http request timeout in seconds for the Elasticsearch request. The default is 90.

flush_interval

edit

The number of seconds to wait for new events between two bulk API index requests. If bulk_max_size is reached before this interval expires, additional bulk index requests are made.

save_topology

edit

A Boolean that specifies whether the topology is kept in Elasticsearch. The default is false.

This option is relevant for Packetbeat only.

topology_expire

edit

The time to live in seconds for the topology information that is stored in Elasticsearch. The default is 15 seconds.

tls

edit

Configuration options for TLS parameters like the certificate authority to use for HTTPS-based connections. If the tls section is missing, the host CAs are used for HTTPS connections to Elasticsearch.

See TLS Options for more information.

Logstash Output

edit

The Logstash output sends the events directly to Logstash by using the lumberjack protocol. To use this option, you must install and configure the logstash-input-beats plugin in Logstash. Logstash allows for additional processing and routing of generated events.

Every event sent to Logstash contains additional metadata for indexing and filtering:

{
    ...
    "@metadata": {
      "beat": "<beat>",
      "type": "<event type>"
    }
}

In Logstash, you can configure the Elasticsearch output plugin to use the metadata and event type for indexing.

The following Logstash 1.5 configuration file sets Logstash to use the index and document type reported by Beats for indexing events into Elasticsearch. The index used will depend on the @timestamp field as identified by Logstash.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    host => "localhost"
    port => "9200"
    protocol => "http"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Here is the same configuration for Logstash 2.x releases:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Events indexed into Elasticsearch with the Logstash configuration shown here will be similar to events directly indexed by Beats into Elasticsearch.

Here is an example of how to configure the Beat to use Logstash:

output:
  logstash:
    hosts: ["localhost:5044"]

    # index configures '@metadata.beat' field to be used by Logstash for
    # indexing. The default index name depends on the each beat.
    # For Packetbeat, the default is set to packetbeat, for Topbeat to
    # topbeat and for Filebeat to filebeat.
    index: topbeat

hosts

edit

The list of known Logstash servers to connect to. All entries in this list can contain a port number. If no port number is given, the value specified for port is used as the default port number.

worker

edit

The number of workers per configured host publishing events to Logstash. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).

loadbalance

edit

If set to true and multiple Logstash hosts are configured, the output plugin load balances published events onto all Logstash hosts. If set to false, the output plugin sends all events to only one host (determined at random) and will switch to another host if the selected one becomes unresponsive. The default value is false.

output:
  logstash:
    hosts: ["localhost:5044", "localhost:5045"]

    # configure logstash plugin to loadbalance events between the logstash instances
    loadbalance: true

    # configure index prefix name
    index: topbeat

port

edit

The default port to use if the port number is not given in hosts. The default port number is 10200.

index

edit

The index root name to write events to. The default is the Beat name. For example "topbeat" generates "[topbeat-]YYYY.MM.DD" indexes (for example, "topbeat-2015.04.26").

tls

edit

Configuration options for TLS parameters like the root CA for Logstash connections. See TLS Options for more information. If the tls section is missing, a TCP-only connection is assumed. Logstash must also be configured to use TCP for Logstash input.

timeout

edit

The number of seconds to wait for responses from the Logstash server before timing out. The default is 30 (seconds).

max_retries

edit

The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. Some Beats, such as Filebeat, ignore the max_retries setting and retry until all events are published.

Set max_retries to a value less than 0 to retry until all events are published.

The default is 3.

Redis Output (DEPRECATED)

edit

The Redis output inserts the events in a Redis list. This output plugin is compatible with the Redis input plugin from Logstash, so the Redis Output for the Beats is deprecated.

Example configuration:

output:
  redis:
    # Set the host and port where to find Redis.
    host: "localhost"
    port: 6379

    # Uncomment out this option if you want to store the topology in Redis.
    # The default is false.
    save_topology: true

    # Optional index name. The default is topbeat and generates topbeat keys.
    index: "topbeat"

    # Optional Redis database number where the events are stored
    # The default is 0.
    db: 0

    # Optional Redis database number where the topology is stored
    # The default is 1. It must have a different value than db.
    db_topology: 1

    # Optional password to authenticate with. By default, no
    # password is set.
    # password: ""

    # Optional Redis initial connection timeout in seconds.
    # The default is 5 seconds.
    timeout: 5

    # Optional interval for reconnecting to failed Redis connections.
    # The default is 1 second.
    reconnect_interval: 1

host

edit

The host of the Redis server.

port

edit

The port of the Redis server.

db

edit

The Redis database number where the events are published. The default is 0.

db_topology

edit

The Redis database number where the topology information is stored. The default is 1.

index

edit

The name of the Redis list where the events are published. The default is "topbeat".

password

edit

The password to authenticate with. The default is no authentication.

timeout

edit

The Redis initial connection timeout in seconds. The default is 5 seconds.

reconnect_interval

edit

The interval for reconnecting failed Redis connections. The default is 1 second.

File Output

edit

The File output dumps the transactions into a file where each transaction is in a JSON format. Currently, this output is used for testing, but it can be used as input for Logstash.

output:

  # File as output
  # Options:
  # path: where to save the files
  # filename: name of the files
  # rotate_every_kb: maximum size of the files in path
  # number of files: maximum number of files in path
  file:
    path: "/tmp/topbeat"
    filename: topbeat
    rotate_every_kb: 1000
    number_of_files: 7

path

edit

The path to the directory where the generated files will be saved. This option is mandatory.

filename

edit

The name of the generated files. The default is set to the Beat name. For example, the files generated by default for Topbeat would be "topbeat", "topbeat.1", "topbeat.2", and so on.

rotate_every_kb

edit

The maximum size in kilobytes of each file. When this size is reached, the files are rotated. The default value is 10240 KB.

number_of_files

edit

The maximum number of files to save under path. When this number of files is reached, the oldest file is deleted, and the rest of the files are shifted from last to first. The default is 7 files.

Console Output

edit

The Console output writes events in JSON format to stdout.

output:
  console:
    pretty: true

pretty

edit

If pretty is set to true, events written to stdout will be nicely formatted. The default is false.

TLS Options

edit

certificate_authorities

edit

The list of root certificates for server verifications. If certificate_authorities is empty or not set, the trusted certificate authorities of the host system are used.

certificate: "/etc/pki/client/cert.pem"

edit

The path to the certificate for TLS client authentication. If the certificate is not specified, client authentication is not available. The connection might fail if the server requests client authentication. If the TLS server does not require client authentication, the certificate will be loaded, but not requested or used by the server.

When this option is configured, the certificate_key: "/etc/pki/client/cert.key" option is also required.

certificate_key: "/etc/pki/client/cert.key"

edit

The client certificate key used for client authentication. This option is required if certificate: "/etc/pki/client/cert.pem" is specified.

min_version

edit

The minimum SSL/TLS version allowed for the encrypted connections. The value must be one of the following: SSL-3.0 for SSL 3, 1.0 for TLS 1.0, 1.1 for TLS 1.1 and 1.2 for TLS 1.2.

The default value is 1.0.

max_version

edit

The maximum SSL/TLS version allowed for the encrypted connections. The value must be one of the following: SSL-3.0 for SSL 3, 1.0 for TLS 1.0, 1.1 for TLS 1.1 and 1.2 for TLS 1.2.

The default value is 1.2.

insecure

edit

This option controls whether the client verifies server certificates and host names. If insecure is set to true, all server host names and certificates are accepted. In this mode, TLS-based connections are susceptible to man-in-the-middle attacks. Use this option for testing only.

cipher_suites

edit

The list of cipher suites to use. The first entry has the highest priority. If this option is omitted, the Go crypto library’s default suites are used (recommended).

Here is a list of allowed cipher suites and their meanings.

  • 3DES: Cipher suites using triple DES
  • AES128/256: Cipher suites using AES with 128/256-bit keys.
  • CBC: Cipher using Cipher Block Chaining as block cipher mode.
  • ECDHE: Cipher suites using Elliptic Curve Diffie-Hellman (DH) ephemeral key exchange.
  • ECDSA: Cipher suites using Elliptic Curve Digital Signature Algorithm for authentication.
  • GCM: Galois/Counter mode is used for symmetric key cryptography.
  • RC4: Cipher suites using RC4.
  • RSA: Cipher suites using RSA.
  • SHA, SHA256, SHA384: Cipher suites using SHA-1, SHA-256 or SHA-384.

The following cipher suites are available:

  • RSA-RC4-128-SHA (disabled by default - RC4 not recommended)
  • RSA-3DES-CBC3-SHA
  • RSA-AES128-CBC-SHA
  • RSA-AES256-CBC-SHA
  • ECDHE-ECDSA-RC4-128-SHA (disabled by default - RC4 not recommended)
  • ECDHE-ECDSA-AES128-CBC-SHA
  • ECDHE-ECDSA-AES256-CBC-SHA
  • ECDHE-RSA-RC4-128-SHA (disabled by default- RC4 not recommended)
  • ECDHE-RSA-3DES-CBC3-SHA
  • ECDHE-RSA-AES128-CBC-SHA
  • ECDHE-RSA-AES256-CBC-SHA
  • ECDHE-RSA-AES128-GCM-SHA256 (TLS 1.2 only)
  • ECDHE-ECDSA-AES128-GCM-SHA256 (TLS 1.2 only)
  • ECDHE-RSA-AES256-GCM-SHA384 (TLS 1.2 only)
  • ECDHE-ECDSA-AES256-GCM-SHA384 (TLS 1.2 only)

curve_types

edit

The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange).

The following elliptic curve types are available:

  • P-256
  • P-384
  • P-521
« Input Shipper »