IMPORTANT: No additional bug fixes or documentation updates
will be released for this version. For the latest information, see the
current release documentation.
Step 2: Exclude all processed files
editStep 2: Exclude all processed files
editFilebeat does not provide access to the state information of different inputs.
Hence, the filestream
input cannot access the state information of a log
input in the
Filebeat registry. You must exclude the files the log
input has processed
or is processing. If you do not exclude those files, you will end up with
duplicate events in the output.
Given the file list and ingestion progress shown earlier,
you should run the log
and filestream
inputs simultaneously until everything
collected by the log
input has made it to the output.
After the files collected by the log
input are shipped and the files are deleted,
you can delete the log
inputs and the exlude_files
settings from filestream
input.
filebeat.inputs: - type: log enabled: true paths: - /var/log/java-exceptions*.log multiline: pattern: '^\[' negate: true match: after close_removed: true close_renamed: true exclude_files: java-exceptions[4-5]{1}.log - type: log enabled: true paths: - /var/log/my-application*.json scan_frequency: 1m json.keys_under_root: true exclude_files: my-application3.log - type: filestream enabled: true id: my-java-collector paths: - /var/log/java-exceptions*.log prospector.scanner.exclude_files: java-exceptions[1-3]{1}.log - type: filestream enabled: true id: my-application-input paths: - /var/log/my-application*.json prospector.scanner.exclude_files: my-application[1-2]{1}.log - type: filestream enabled: true id: my-old-files paths: - /var/log/my-old-files*.log