Introducing Filestream fingerprint mode

Filestream is an input type in Filebeat that is used for ingesting files from the given paths. 

To explain what fingerprint mode is and where exactly we introduced it in Filestream, let’s start by explaining the basic architecture of a Filestream input:

By default, File Scanner is using file system metadata to compare files when searching for renames/moves such as: <inode>-<device_id> string on Unix systems (more about inode can be found here) and <idxhi>-<idxlo>-<vol> string on Windows (nFileIndexHigh, nFileIndexLow and dwVolumeSerialNumber respectively — read more in the official documentation from Microsoft). The same string is used as a unique file identifier returned by File Identity Provider, and this value is used as a key for each file in Registry to look up the file’s current state.

The whole point of the unique file identifier returned by the File Identity Provider is that it must be stable, meaning it won’t change during the time the file is being ingested by Filestream. It must be stable because Filestream uses this identifier to track file metadata, including the current offset of the file, so it knows where to continue the ingestion.

What if the identifier is not stable? It leads to data loss or data duplication.

Example of the data loss:

• A file ID now matches a different file (not ingested before).
• Instead of reading this file from offset 0, as Filestream is supposed to, the wrong offset information is applied to this file.
• Filestream continues reading the log lines too far forward in the file, skipping log lines. These lines will never make it to the output.

Example of the data duplication:

• A file ID has changed for an existing file.
• It now appears as a new file to Filestream.
• Filestream starts reading it from offset 0 (re-ingestion).

Unfortunately, not all file systems can produce stable device_id and inode values.

If you try to run this script on different file systems, you might see different results:

Ext file systems (e.g., ext4) store the inode number in the i_ino file, inside struct inode, which is written to disk. In this case, if the file is the same (not another file with the same name) then the inode number is guaranteed to be the same.

If the file system is other than Ext, the inode number is generated by the inode operations defined by the filesystem driver. As they don't have the concept of what an inode is, they have to mimic all inode's internal fields to comply with VFS, so this number will probably be different after reboot — in theory, even after closing and opening the file again.

Sources: 

• https://stackoverflow.com/a/21123293 
• https://stackoverflow.com/a/4677892 

• We’ve seen our customers experiencing problems with using rsync and changed inodes.
• Also, not everyone knows that sed -i creates a temporary file and then moves it to the place of the original file changing the inode value (it’s basically a new file). For example, some users might use sed -i for masking credentials in logs.

In addition to problems with inode, depending on how disk drives are mounted, it’s possible that the device_id can change after a reboot. However, we’ve already introduced a solution for this problem some time ago: file_identity: inodemarker.

The new fingerprint mode is implemented in the File Scanner component to avoid the above-mentioned problems.

The new fingerprint mode switches the default File Scanner behavior from using file system metadata to using SHA256 hash for a given byte range of the file. By default this range is from 0 to 1024 but it’s configurable by offset and length config parameters.

There are some points one must consider before starting to use this new feature:

• All your logs files must be unique in the configured byte range. Most of the log files are, due to timestamps and just the pure nature of logs, but one must inspect the logs and decide on the offset and length of the fingerprint.
• Once you start using file_identity: fingerprint, you cannot change the offset and length of the fingerprint anymore; it would lead to a total reingestion of all the files that match the paths of the input.
• There is a performance hit — the performance aspect of this feature deserves a separate section in this article:

From the very early stage of working on this feature, there was a concern regarding the performance hit it would cause in File Scanner. In the end, we would need to open a file, read the amount of bytes set by the prospector.scanner.fingerprint.length config option, and compute a SHA256 out of it. And we would need to do that for every single file that matches the globs in paths.

A slight side note here: in order to implement this new feature, File Scanner had to be changed a lot. So, when I worked on the code, I took this opportunity to refactor some parts of File Scanner and make some optimizations while doing so, mainly reducing the amount of syscalls. I also added a lot of tests to verify the expected behavior. So, there was a suspicion that the new File Scanner (fingerprint mode disabled) is faster since it does not make as many syscalls anymore.

After the fingerprint mode was finally delivered to main, I ran some benchmarks, and the results were interesting to say the least:

This new fingerprint mode solves a lot of issues with unstable metadata on file systems and, compared to the previous version of Filebeat, it’s even faster than the default mode.

Also, the default mode got much faster in the new Filebeat release, so it seems very beneficial to refactor some old code from time to time and run benchmarks/profiling to see how the performance changed.

We’ll continue to focus on the performance of Filebeat. Stay tuned.

What else is new in Elastic 8.10? Check out the 8.10 announcement post to learn more.