Frequently asked questions

This section contains frequently asked questions about Filebeat. Also check out the Filebeat discussion forum.

We do not recommend reading log files from network volumes. Whenever possible, install Filebeat on the host machine and send the log files directly from there. Reading files from network volumes (especially on Windows) can have unexpected side effects. For example, changed file identifiers may result in Filebeat reading a log file from scratch again.

Filebeat might be incorrectly configured or unable to send events to the output. To resolve the issue:

Filebeat keeps the file handler open in case it reaches the end of a file so that it can read new log lines in near real time. If Filebeat is harvesting a large number of files, the number of open files can become an issue. In most environments, the number of files that are actively updated is low. The close_inactive configuration option should be set accordingly to close files that are no longer active.

There are additional configuration options that you can use to close file handlers, but all of them should be used carefully because they can have side effects. The options are:

The close_renamed and close_removed options can be useful on Windows to resolve issues related to file rotation. See Open file handlers cause issues with Windows file rotation?. The close_eof option can be useful in environments with a large number of files that have only very few entries. The close_timeout option is useful in environments where closing file handlers is more important than sending all log lines. For more details, see Set up prospectors.

Make sure that you read the documentation for these configuration options before using any of them.

Filebeat keeps the state of each file and persists the state to disk in the registry_file. The file state is used to continue file reading at a previous position when Filebeat is restarted. If a large number of new files are produced every day, the registry file might grow to be too large. To reduce the size of the registry file, there are two configuration options available: clean_removed and clean_inactive.

For old files that you no longer touch and are ignored (see ignore_older), we recommended that you use clean_inactive. If old files get removed from disk, then use the clean_removed option.

On Linux file systems, Filebeat uses the inode and device to identify files. When a file is removed from disk, the inode may be assigned to a new file. In use cases involving file rotation, if an old file is removed and a new one is created immediately afterwards, the new file may have the exact same inode as the file that was removed. In this case, Filebeat assumes that the new file is the same as the old and tries to continue reading at the old position, which is not correct.

By default states are never removed from the registry file. To resolve the inode reuse issue, we recommend that you use the clean_* options, especially clean_inactive, to remove the state of inactive files. For example, if your files get rotated every 24 hours, and the rotated files are not updated anymore, you can set ignore_older to 48 hours and clean_inactive to 72 hours.

You can use clean_removed for files that are removed from disk. Be aware that clean_removed cleans the file state from the registry whenever a file cannot be found during a scan. If the file shows up again later, it will be sent again from scratch.

On Windows, you might have problems renaming or removing files because Filebeat keeps the file handlers open. This can lead to issues with the file rotating system. To avoid this issue, you can use the close_removed and close_renamed options together.

Filebeat might be configured to scan for files too frequently. Check the setting for scan_frequency in the filebeat.yml config file. Setting scan_frequency to less than 1s may cause Filebeat to scan the disk in a tight loop.

The index template might not be loaded correctly. See Step 4: Load the index template in Elasticsearch.

If you have recently performed an operation that loads or parses custom, structured logs, you might need to refresh the index to make the fields available in Kibana. To refresh the index, use the refresh API. For example:

curl -XPOST 'http://localhost:9200/filebeat-2016.08.09/_refresh'

Filebeat uses a newline character to detect the end of an event. If lines are added incrementally to a file that’s being harvested, a newline character is required after the last line, or Filebeat will not read the last line of the file.

In the default behaviour, Filebeat opens the files and keeps them open until it reaches the end of them. In situations when the configured output is blocked (e.g. Elasticsearch or Logstash is unavailable) for a long time, this can cause Filebeat to keep file handlers to files that were deleted from the file system in the mean time. As long as Filebeat keeps the deleted files open, the operating system doesn’t free up the space on disk, which can lead to increase disk utilisation or even out of disk situations.

To mitigate this issue, you can set the close_timeout setting to 5m. This will ensure every file handler is closed once every 5 minutes, regardless of whether it reached EOF or not. Note that this option can lead to data loss if the file is deleted before Filebeat reaches the end of the file.

If you need to limit bandwidth usage, we recommend that you configure the network stack on your OS to perform bandwidth throttling.

For example, the following Linux commands cap the connection between Filebeat and Logstash by setting a limit of 50 kbps on TCP connections over port 5044:

tc qdisc add dev $DEV root handle 1: htb
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 50kbps ceil 50kbps
tc filter add dev $DEV parent 1:0 prio 1 protocol ip handle 10 fw flowid 1:10
iptables -A OUTPUT -t mangle -p tcp --dport 5044 -j MARK --set-mark 10

Using OS tools to perform bandwidth throttling gives you better control over policies. For example, you can use OS tools to cap bandwidth during the day, but not at night. Or you can leave the bandwidth uncapped, but assign a low priority to the traffic.

You may encounter errors loading the config file on POSIX operating systems if:

See Config File Ownership and Permissions for more about resolving these errors.

Either there is a problem with the structure of your config file, or you have used a path or expression that the YAML parser cannot resolve because the config file contains characters that aren’t properly escaped.

If the YAML file contains paths with spaces or unusual characters, wrap the paths in single quotation marks (see Wrap paths in single quotation marks).

Also see the general advice under YAML tips and gotchas.

You may have configured Logstash or Filebeat incorrectly. To resolve the issue:

Logstash outputs remove @metadata fields automatically. Therefore, if Logstash instances are chained directly or via some message queue (for example, Redis or Kafka), the @metadata field will not be available in the final Logstash instance.

Beats are lightweight data shippers that you install as agents on your servers to send specific types of operational data to Elasticsearch. Beats have a small footprint and use fewer system resources than Logstash.

Logstash has a larger footprint, but provides a broad array of input, filter, and output plugins for collecting, enriching, and transforming data from a variety of sources.

For more information, see the Logstash Introduction and the Beats Overview.

The host running Logstash might be unreachable or the certificate may not be valid. To resolve your issue:

Here are some common errors and ways to fix them:

This happens because your certificate is only valid for the hostname present in the Subject field.

To resolve this problem, try one of these solutions:

This is not a SSL problem. It’s a networking problem. Make sure the two hosts can communicate.

This is not a SSL problem. Make sure that Logstash is running and that there is no firewall blocking the traffic.

A firewall is refusing the connection. Check if a firewall is blocking the traffic on the client, the network, or the destination host.