WARNING: Version 5.6 of Filebeat has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

› › ›

Tutorial

edit

Tutorial

edit

This tutorial assumes you have Elasticsearch and Kibana installed and accessible from Filebeat (see the getting started section). It also assumes that the Ingest Node GeoIP and User Agent plugins are installed. These plugins are required to capture the geographical location and browser information used by some of the visualizations available in the sample dashboards. You can install these plugins by running the following commands in the Elasticsearch home path:

sudo bin/elasticsearch-plugin install ingest-geoip
sudo bin/elasticsearch-plugin install ingest-user-agent

You need to restart Elasticsearch after running these commands.

If you are using an Elastic Cloud instance, you can enable the two plugins from the configuration page.

This also assumes you have Nginx installed and writing logs in the default location and format. If you want to monitor another service for which a module exists, adjust the commands in the tutorial accordingly.

You can start Filebeat with the following command:

./filebeat -e -modules=nginx -setup

The -e flag tells Filebeat to output its logs to standard error, instead of syslog.

The -modules=nginx flag loads the Nginx module.

The -setup flag tells Filebeat to load the associated sample Kibana dashboards. This setup phase, in which the dashboards are loaded, doesn’t have to be executed each time, and because it’s a relatively heavy operation, we recommend executing it only once after installing or upgrading Filebeat. That is why, the next commands from this tutorial are omitting the -setup flag.

Visiting the Kibana web interface now, open the Nginx dashboard and you should already see your logs parsed and visualized in several widgets.

kibana nginx

You can also start multiple modules at once:

./filebeat -e -modules=nginx,mysql,system

Because Filebeat modules are currently in Beta, the default Filebeat configuration may interfere with the Filebeat system module configuration. If you plan to run the system module, edit the Filebeat configuration file, filebeat.yml, and comment out the following lines:

#- input_type: log
  #paths:
    #- /var/log/*.log

For rpm and deb, you’ll find the configuration file at /etc/filebeat/filebeat.yml. For mac and win, look in the archive that you extracted when you installed Filebeat.

While enabling the modules from the CLI file is handy for getting started and for testing, you will probably want to use the configuration file for the production setup. The equivalent of the above in the configuration file is:

filebeat.modules:
- module: nginx
- module: mysql
- module: system

Then you can start Filebeat simply with: ./filebeat -e.

Variable overrides

edit

Each module and fileset has a set of "variables" which allow adjusting their behaviour. To see the available variables, you can consult the filebeat.full.yml file. For example, all filesets allow setting a custom paths value, which is a list of Globs where the log files are searched.

These variables have default values, sometimes depending on the operating system. You can override them either from the CLI via the -M flag, or from the configuration file.

In the case of Nginx, for example, you can use the following if the access files are in a custom location:

./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]"

Or via the configuration file:

filebeat.modules:
- module: nginx
  access:
    var.paths: ["/var/log/nginx/access.log*"]

Advanced settings

edit

Behind the scenes, each module starts a Filebeat prospector. For advanced users, it’s possible to add or overwrite any of the prospector settings. For example, enabling close_eof can be done like this:

filebeat.modules:
- module: nginx
  access:
    prospector:
      close_eof: true

Or like this:

./filebeat -e -modules=nginx -M "nginx.access.prospector.close_eof=true"

From the CLI, it’s possible to change variables or settings for multiple modules/fileset at once. For example, the following works and will enable close_eof for all the filesets in the nginx module:

./filebeat -e -modules=nginx -M "nginx.*.prospector.close_eof=true"

The following also works and will enable close_eof for all prospectors created by any of the modules:

./filebeat -e -modules=nginx,mysql -M "*.*.prospector.close_eof=true"

« Modules overview Apache2 module »