Specifying Multiple Prospectors

When you need to collect lines from multiple files, you can simply configure a single prospector and specify multiple paths to start a harvester for each file. However, if you want to apply additional prospector-specific configuration settings (such as fields, include_lines, exclude_lines, multiline, and so on) to the lines harvested from specific files, you need to define multiple prospectors in the Filebeat config file.

Within the config file, you can specify multiple prospectors, and each prospector can define multiple paths to crawl, as shown in the following example.

filebeat:
  prospectors:
    -
      paths:
        - /var/log/system.log
        - /var/log/wifi.log
    -
      paths:
        - "/var/log/apache2/*"
      fields:
        apache: true
      fields_under_root: true

The config file in the example starts two prospectors (the list of prospectors is a YAML array, so each prospector begins with a -). The first prospector has two harvesters, one harvesting the system.log file, and the other harvesting wifi.log. The second prospector starts a harvester for each file in the apache2 directory and uses the fields configuration option to add a field called apache to the output.