Step 2: Exclude all processed files

Filebeat does not provide access to the state information of different inputs. Hence, the filestream input cannot access the state information of a log input in the Filebeat registry. You must exclude the files the log input has processed or is processing. If you do not exclude those files, you will end up with duplicate events in the output.

Given the file list and ingestion progress shown earlier, you should run the log and filestream inputs simultaneously until everything collected by the log input has made it to the output. After the files collected by the log input are shipped and the files are deleted, you can delete the log inputs and the exlude_files settings from filestream input.

filebeat.inputs:
 - type: log
   enabled: true
   paths:
     - /var/log/java-exceptions*.log
   multiline:
    pattern: '^\['
    negate: true
    match: after
  close_removed: true
  close_renamed: true
  exclude_files: java-exceptions[4-5]{1}.log

- type: log
  enabled: true
  paths:
    - /var/log/my-application*.json
  scan_frequency: 1m
  json.keys_under_root: true
  exclude_files: my-application3.log

- type: filestream
  enabled: true
  id: my-java-collector
  paths:
    - /var/log/java-exceptions*.log
  prospector.scanner.exclude_files: java-exceptions[1-3]{1}.log

- type: filestream
  enabled: true
  id: my-application-input
  paths:
    - /var/log/my-application*.json
  prospector.scanner.exclude_files: my-application[1-2]{1}.log

- type: filestream
  enabled: true
  id: my-old-files
  paths:
    - /var/log/my-old-files*.log

« Step 1: Set an identifier for each filestream input Step 3: Use new option names »