IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Converting Ingest Node Pipelines Managing Logstash »

› ›

Configuring Logstash-to-Logstash Communication

edit

Configuring Logstash-to-Logstash Communication

edit

You can set up communication between two Logstash machines by connecting the Lumberjack output to the Beats input. You may need this configuration if the data path crosses network or firewall boundaries, for example. If you don’t have a compelling need for Logstash-to-Logstash communication, then don’t implement it.

Configuration overview

edit

Use the Lumberjack protocol to connect two Logstash machines.

Generate a trusted SSL certificate (required by the lumberjack protocol).
Copy the SSL certificate to the upstream Logstash machine.
Copy the SSL certificate and key to the downstream Logstash machine.
Set the upstream Logstash machine to use the Lumberjack output to send data.
Set the downstream Logstash machine to listen for incoming Lumberjack connections through the Beats input.
Test it.

Generate a self-signed SSL certificate and key

edit

Use the openssl req command to generate a self-signed certificate and key. The openssl req command is available with some operating systems. You may need to install the openssl command line program for others.

Run the following command:

openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.cert -subj /CN=localhost

where:

lumberjack.key is the name of the SSL key to be created
lumberjack.cert is the name of the SSL certificate to be created
localhost is the name of the upstream Logstash computer

This command produces output similar to the following:

Generating a 2048 bit RSA private key
.................................+++
....................+++
writing new private key to 'lumberjack.key'

Copy the SSL certificate and key

edit

Copy the SSL certificate to the upstream Logstash machine.

Copy the SSL certificate and key to the downstream Logstash machine.

Start the upstream Logstash instance

edit

Start Logstash and generate test events:

bin/logstash -e 'input { generator { count => 5 } } output { lumberjack { codec => json hosts => "mydownstreamhost" ssl_certificate => "lumberjack.cert" port => 5000 } }'

This sample command sends five events to mydownstreamhost:5000 using the SSL certificate provided.

Start the downstream Logstash instance

edit

Start the downstream instance of Logstash:

bin/logstash -e 'input { beats { codec => json port => 5000 ssl => true ssl_certificate => "lumberjack.cert" ssl_key => "lumberjack.key"} }'

This sample command sets port 5000 to listen for incoming Beats input.

Verify the communication

edit

Watch the downstream Logstash machine for the incoming events. You should see five incrementing events similar to the following:

{
  "@timestamp" => 2018-02-07T12:16:39.415Z,
  "sequence"   => 0
  "tags"       => [
    [0] "beats_input_codec_json_applied"
  ],
  "message"    => "Hello world",
  "@version"   => "1",
  "host"       => "ls1.semicomplete.com"
}

If you see all five events with consistent fields and formatting, incrementing by one, then your configuration is correct.

« Converting Ingest Node Pipelines Managing Logstash »