WARNING: Version 1.0.1 of Beats has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Installing Elasticsearch Installing Kibana »

› ›

Installing Logstash (Optional)

edit

Installing Logstash (Optional)

edit

The simplest architecture for the Beats platform setup consists of one or more Beats, Elasticsearch, and Kibana. This architecture is easy to get started with and sufficient for networks with low traffic. It also uses the minimum amount of servers: a single machine running Elasticsearch and Kibana. The Beats insert the transactions directly into the Elasticsearch instance.

This section explains how to use the Beats together with Logstash to provide additional buffering. An important advantage to this approach is that you can use Logstash to modify the data captured by Beats in any way you like. You can also use Logstash’s many output plugins to integrate with other systems.

Integration with Logstash

To download and install Logstash, use the commands that work with your system:

deb:

sudo apt-get install openjdk-7-jre
curl -L -O https://download.elastic.co/logstash/logstash/packages/debian/logstash_2.1.1-1_all.deb
sudo dpkg -i logstash_2.1.1-1_all.deb

rpm:

sudo yum install java-1.7.0-openjdk
curl -L -O https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.1.1-1.noarch.rpm
sudo rpm -i logstash-2.1.1-1.noarch.rpm

mac:

# install Java, e.g. from: https://www.java.com/en/download/manual.jsp
curl -L -O https://download.elastic.co/logstash/logstash/logstash-2.1.1.zip
unzip logstash-2.1.1.zip

win:

If necessary, download and install the latest version of the Java from www.java.com.
Download the Logstash 2.1.1 Windows zip file from the downloads page.
Extract the contents of the zip file to a directory on your computer, for example, C:\Program Files.

Don’t start Logstash yet. You need to set a couple of configuration options first.

Setting Up Logstash

edit

In this setup, the Beat sends events to Logstash. Logstash receives these events by using the Logstash Input Beats plugin and then sends the transaction to Elasticsearch by using the Elasticsearch output plugin. The Elasticsearch plugin of Logstash uses the bulk API, making indexing very efficient.

The minimum required Logstash version for this plugin is 1.5.4. If you are using Logstash 1.5.4, you must install the Beats input plugin before applying this configuration because the plugin is not shipped with 1.5.4. To install the required plugin, run the following command inside the logstash directory (for deb and rpm installs, the directory is /opt/logstash).

deb, rpm, and mac:

./bin/plugin install logstash-input-beats

win:

bin\plugin install logstash-input-beats

Next configure Logstash to listen on port 5044 for incoming Beats connections and to index into Elasticsearch. You configure Logstash by creating a configuration file. For example, you can save the following example configuration to a file called config.json:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Logstash uses this configuration to index events in Elasticsearch in the same way that the Beat would, but you get additional buffering and other capabilities provided by Logstash.

To use this setup, you’ll also need to configure your Beat to use Logstash. For more information, see the documentation for your Beat.

Updating the Logstash Input Beats Plugin

edit

Plugins have their own release cycle and are often released independent of Logstash’s core release cycle. To ensure that you have the latest version of the Logstash Input Beats plugin, run the following command from your Logstash installation:

deb, rpm, and mac:

./bin/plugin update logstash-input-beats

win:

bin\plugin update logstash-input-beats

Keep in mind that you can update to the latest version of the plugin without having to upgrade to a newer version of Logstash. More details about working with input plugins in Logstash are available here.

Running Logstash

edit

Now you can start Logstash. Use the command that works with your system. If you installed Logstash as a deb or rpm package, make sure the config file is in the expected directory.

deb:

sudo /etc/init.d/logstash start

rpm:

sudo service logstash start

mac:

./bin/logstash -f config.json

win:

bin\logstash.bat -f config.json

The default configuration for Beats and Logstash uses plain TCP. For encryption you must explicitly enable TLS when you configure Beats and Logstash.

You can learn more about installing, configuring, and running Logstash here.

« Installing Elasticsearch Installing Kibana »