WARNING: Version 1.0.1 of Packetbeat has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Interfaces Maintaining the Real-Time State of the Network Topology »

› ›

Protocols

edit

Protocols

edit

The protocols section contains configuration options for each supported protocol, including common options like ports, send_request, send_response, and options that are protocol-specific.

Currently, Packetbeat supports the following protocols:

DNS
HTTP
Mysql
PostgreSQL
Redis
Thrift-RPC
MongoDB
Memcache

Example configuration:

protocols:
  dns:
    ports: [53]

  http:
    ports: [80, 8080, 8000, 5000, 8002]

  memcache:
    ports: [11211]

  mysql:
    ports: [3306]

  redis:
    ports: [6379]

  pgsql:
    ports: [5432]

  thrift:
    ports: [9090]

Common Protocol Options

edit

The following options are available for all protocols:

ports

edit

The ports where Packetbeat will look to capture traffic for specific protocols. Packetbeat installs a BPF filter based on the ports specified in this section. If a packet doesn’t match the filter, very little CPU is required to discard the packet. Packetbeat also uses the ports specified here to determine which parser to use for each packet.

send_request

edit

If this option is enabled, the raw message of the request (request field) is sent to Elasticsearch. The default is false. This option is useful when you want to index the whole request. Note that for HTTP, the body is not included by default, only the HTTP headers.

send_response

edit

If this option is enabled, the raw message of the response (response field) is sent to Elasticsearch. The default is false. This option is useful when you want to index the whole response. Note that for HTTP, the body is not included by default, only the HTTP headers.

transaction_timeout

edit

The per protocol transaction timeout. Expired transactions will no longer be correlated to incoming responses, but sent to Elasticsearch immediately.

DNS Configuration Options

edit

The dns section specifies configuration options for the DNS protocol. The DNS protocol supports processing DNS messages on UDP. Here is a sample configuration section for DNS:

protocols:
  dns:
    ports: [53]

    # include_authorities controls whether or not the dns.authorities field
    # (authority resource records) is added to messages.
    # Default: false
    include_authorities: true
    # include_additionals controls whether or not the dns.additionals field
    # (additional resource records) is added to messages.
    # Default: false
    include_additionals: true

include_authorities

edit

If this option is enabled, dns.authority fields (authority resource records) are added to DNS events. The default is false.

include_additionals

edit

If this option is enabled, dns.additionals fields (additional resource records) are added to DNS events. The default is false.

HTTP Configuration Options

edit

The HTTP protocol has several specific configuration options. Here is a sample configuration section:

protocols:
  http:

    # Configure the ports where to listen for HTTP traffic. You can disable
    # the http protocol by commenting the list of ports.
    ports: [80, 8080, 8000, 5000, 8002]

    # Uncomment the following to hide certain parameters in URL or forms attached
    # to HTTP requests. The names of the parameters are case insensitive.
    # The value of the parameters will be replaced with the 'xxxxx' string.
    # This is generally useful for avoiding storing user passwords or other
    # sensitive information.
    hide_keywords: ["pass", "password", "passwd"]

    # Uncomment the following to export a list of extra HTTP headers. By
    default is none sent.
    send_headers: ["User-Agent", "Cookie", "Set-Cookie"]

    # Uncomment the following to export Cookie or Set-Cookie headers. By
    # default is false.
    split_coookie: true

    # Configure the HTTP header that contains the real IP address.
    real_ip_header: "X-Forwarded-For"

hide_keywords

edit

A list of query parameters that Packetbeat will automatically censor in the transactions that it saves. The values associated with these parameters are replaced by 'xxxxx'. By default, no changes are made to the HTTP messages.

Packetbeat has this option because, unlike SQL traffic, which typically only contains the hashes of the passwords, HTTP traffic may contain sensitive data. To reduce security risks, you can configure this option to avoid sending the contents of certain HTTP POST parameters.

This option replaces query parameters from GET requests and top-level parameters from POST requests. If sensitive data is encoded inside a parameter that you don’t specify here, Packetbeat cannot censor it. Also, note that if you configure Packetbeat to save the raw request and response fields (see the send_request and the send_response options), sensitive data may be present in those fields.

redact_authorization

edit

When this option is enabled, Packetbeat obscures the value of Authorization and Proxy-Authorization HTTP headers, and censors those strings in the response.

You should set this option to true for transactions that use Basic Authentication because they may contain the base64 unencrypted username and password.

send_headers

edit

A list of header names to capture and send to Elasticsearch. These headers are placed under the headers dictionary in the resulting JSON.

send_all_headers

edit

Instead of sending a white list of headers to Elasticsearch, you can send all headers by setting this option to true. The default is false.

include_body_for

edit

The list of content types for which Packetbeat includes the full HTTP payload in the response field. This option should be used together with the send_response option.

Example configuration:

protocols:
  http:
    ports: [80, 8080]
    send_response: true
    include_body_for: ["text/html"]

split_cookie

edit

If the Cookie or Set-Cookie headers are sent, this option controls whether they are split into individual values. For example, with this option set, an HTTP response might result in the following JSON:

"response": {
  "code": 200,
  "headers": {
    "connection": "close",
    "content-language": "en",
    "content-type": "text/html; charset=utf-8",
    "date": "Fri, 21 Nov 2014 17:07:34 GMT",
    "server": "gunicorn/19.1.1",
    "set-cookie": { 
      "csrftoken": "S9ZuJF8mvIMT5CL4T1Xqn32wkA6ZSeyf",
      "expires": "Fri, 20-Nov-2015 17:07:34 GMT",
      "max-age": "31449600",
      "path": "/"
    },
    "vary": "Cookie, Accept-Language"
  },
  "phrase": "OK"
}

Note that set-cookie is a map containing the cookie names as keys.

The default is false.

real_ip_header

edit

The header field to extract the real IP from. This setting is useful when you want to capture traffic behind a reverse proxy, but you want to get the geo-location information. If this header is present and contains a valid IP addresses, the information is used for the real_ip and client_location indexed fields.

Memcache Configuration Options

edit

The memcache section specifies configuration options for the memcache protocol. Here is a sample configuration section for memcache:

  memcache:
    ports: [11211]
    parseunknown: false
    maxvalues: 0
    maxbytespervalue: 100
    transaction_timeout: 200
    udptransactiontimeout: 200

parseunknown

edit

When this option is enabled, it forces the memcache text protocol parser to accept unknown commands.

The unknown commands MUST NOT contain a data part.

maxvalues

edit

The maximum number of values to store in the message (multi-get). All values will be base64 encoded.

The possible settings for this option are:

maxvalue: -1, which stores all values (text based protocol multi-get)
maxvalue: 0, which stores no values (default)
maxvalue: N, which stores up to N values

maxbytespervalue

edit

The maximum number of bytes to be copied for each value element.

Values will be base64 encoded, so the actual size in the JSON document will be 4 times the value that you specify for maxbytespervalue.

udptransactiontimeout

edit

The transaction timeout in milliseconds. The defaults is 10000 milliseconds.

Quiet messages in UDP binary protocol get responses only if there is an error. The memcache protocol analyzer will wait for the number of milliseconds specified by udptransactiontimeout before publishing quiet messages. Non-quiet messages or quiet requests with an error response are published immediately.

MySQL and PgSQL Configuration Options

edit

max_rows

edit

The maximum number of rows from the SQL message to publish to Elasticsearch. The default is 10 rows.

max_row_length

edit

The maximum length in bytes of a row from the SQL message to publish to Elasticsearch. The default is 1024 bytes.

Thrift Configuration Options

edit

The Thrift protocol has several specific configuration options. Here is a sample configuration section:

  thrift:
    transport_type: socket
    protocol_type: binary
    idl_files: ["tutorial.thrift", "shared.thrift"]
    string_max_size: 200
    collection_max_size: 20
    capture_reply: true
    obfuscate_strings: true
    drop_after_n_struct_fields: 100

transport_type

edit

The Thrift transport type. Currently this option accepts the values socket for TSocket, which is the default Thrift transport, and framed for the TFramed Thrift transport. The default is socket.

protocol_type

edit

The Thrift protocol type. Currently the only accepted value is binary for the TBinary protocol, which is the default Thrift protocol.

idl_files

edit

The Thrift interface description language (IDL) files for the service that Packetbeat is monitoring. Providing the IDL files is optional, because the Thrift messages contain enough information to decode them without having the IDL files. However, providing the IDL enables Packetbeat to include parameter and exception names.

string_max_size

edit

The maximum length for strings in parameters or return values. If a string is longer than this value, the string is automatically truncated to this length. Packetbeat adds dots at the end of the string to mark that it was truncated. The default is 200.

collection_max_size

edit

The maximum number of elements in a Thrift list, set, map, or structure. If a collection has more elements than this value, Packetbeat captures only the specified number of elements. Packetbeat adds a fictive last element ... to the end of the collection to mark that it was truncated. The default is 15.

capture_reply

edit

If this option is set to false, Packetbeat decodes the method name from the reply and simply skips the rest of the response message. This setting can be useful for performance, disk usage, or data retention reasons. The default is true.

obfuscate_strings

edit

If this option is set to true, Packetbeat replaces all strings found in method parameters, return codes, or exception structures with the "*" string.

drop_after_n_struct_fields

edit

The maximum number of fields that a structure can have before Packetbeat ignores the whole transaction. This is a memory protection mechanism (so that Packetbeat’s memory doesn’t grow indefinitely), so you would typically set this to a relatively high value. The default is 500.

MongoDB Configuration

edit

The following settings are specific to the MongoDB protocol. Here is a sample configuration section:

  mongodb:
    send_request: true
    send_response: true
    max_docs: 0
    max_doc_length: 0

The following two settings are useful for limiting the amount of data Packetbeat indexes in the response fields.

max_docs

edit

The maximum number of documents from the response to index in the response field. The default is 10. You can set this to 0 to index an unlimited number of documents.

Packetbeat adds a [...] line at the end to signify that there were additional documents that weren’t saved because of this setting.

max_doc_length

edit

The maximum number of characters in a single document indexed in the response field. The default is 5000. You can set this to 0 to index an unlimited number of characters per document.

If the document is trimmed because of this setting, Packetbeat adds the string ... at the end of the document.

Note that limiting documents in this way means that they are no longer correctly formatted JSON objects.

« Interfaces Maintaining the Real-Time State of the Network Topology »