IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Logstash-to-Logstash: Lumberjack output to Beats input Managing Logstash »

› ›

Logstash-to-Logstash: HTTP output to HTTP input

edit

Logstash-to-Logstash: HTTP output to HTTP input

edit

HTTP output to HTTP input is an alternative to the Lumberjack output to Beats input approach for Logstash-to-Logstash communication. This approach relies on the use of http output to http input plugins.

Check out these considerations before you implement Logstash-to-Logstash using HTTP.

For now, http output to http input with manual configuration may be the best path forward if these limitations don’t apply to your use case.

Configuration overview

edit

To use the HTTP protocol to connect two Logstash instances:

Configure the downstream (server) Logstash to use HTTP input
Configure the upstream (client) Logstash to use HTTP output
Secure the communication between HTTP input and HTTP output

Configure the downstream Logstash to use HTTP input

edit

Configure the HTTP input on the downstream (receiving) Logstash to receive connections. The minimum configuration requires these options:

port - To set a custom port.
additional_codecs - To set application/json to be json_lines.

input {
    http {
        port => 8080
        additional_codecs => { "application/json" => "json_lines" }
    }
}

Configure the upstream Logstash to use HTTP output

edit

In order to obtain the best performance when sending data from one Logstash to another, the data needs to be batched and compressed. As such, the upstream Logstash (the sending Logstash) needs to be configured with these options:

url - The receiving Logstash.
http_method - Set to post.
retry_non_idempotent - Set to true, in order to retry failed events.
format - Set to json_batch to batch the data.
http_compression - Set to true to ensure the data is compressed.

output {
    http {
        url => '<protocol>://<downstream-logstash>:<port>'
        http_method => post
        retry_non_idempotent => true
        format => json_batch
        http_compression => true
    }
}

Secure Logstash to Logstash

edit

It is important that you secure the communication between Logstash instances. Use SSL/TLS mutual authentication in order to ensure that the upstream Logstash instance sends encrypted data to a trusted downstream Logstash instance, and vice versa.

Create a certificate authority (CA) in order to sign the certificates that you plan to use between Logstash instances. Creating a correct SSL/TLS infrastructure is outside the scope of this document.

We recommend you use the elasticsearch-certutil tool to generate your certificates.
Configure the downstream (receiving) Logstash to use SSL. Add these settings to the HTTP Input configuration:
- ssl: When set to true, it enables Logstash use of SSL/TLS
- ssl_key: Specifies the key that Logstash uses to authenticate with the client.
- ssl_certificate: Specifies the certificate that Logstash uses to authenticate with the client.
- ssl_certificate_authorities: Configures Logstash to trust any certificates signed by the specified CA.
- ssl_verify_mode: Specifies whether Logstash server verifies the client certificate against the CA.
For example:
```
input {
  http {
    ...

    ssl => true
    ssl_key => "server.key.pk8"
    ssl_certificate => "server.crt"
    ssl_certificate_authorities => "ca.crt"
    ssl_verify_mode => force_peer
  }
}
```
Configure the upstream (sending) Logstash to use SSL. Add these settings to the HTTP output configuration:
- cacert: Configures the Logstash client to trust any certificates signed by the specified CA.
- client_key: Specifies the key the Logstash client uses to authenticate with the Logstash server.
- client_cert: Specifies the certificate that the Logstash client uses to authenticate to the Logstash server.
For example:
```
output {
  http {
    ...

    cacert => "ca.crt"
    client_key => "client.key.pk8"
    client_cert => "client.crt"
  }
}
```
If you would like an additional authentication step, you can also use basic user/password authentication in both Logstash instances:
- user: Sets the username to use for authentication.
- password: Sets the password to use for authentication.
For example, you would need to add the following to both Logstash instances:
```
...
  http {
    ...

    user => "your-user"
    password => "your-secret"
  }
...
```

« Logstash-to-Logstash: Lumberjack output to Beats input Managing Logstash »