Frequently Asked Questions

This section contains frequently asked questions about Filebeat. Also check out the Filebeat discussion forum.

We do not recommend reading log files from network volumes. Whenever possible, install Filebeat on the host machine and send the log files directly from there. Reading files from network volumes (especially on Windows) can have unexpected side effects. For example, changed file identifiers may result in Filebeat reading a log file from scratch again.

Filebeat might be incorrectly configured or unable to send events to the output. To resolve the issue:

Filebeat might be configured to scan for files too frequently. Check the setting for scan_frequency in the filebeat.yml config file. Setting scan_frequency to less than 1s may cause Filebeat to scan the disk in a tight loop.

The index template might not be loaded correctly. See Step 4: Loading the Index Template in Elasticsearch.

Filebeat uses a newline character to detect the end of an event. If lines are added incrementally to a file that’s being harvested, a newline character is required after the last line, or Filebeat will not read the last line of the file.

If you need to limit bandwidth usage, we recommend that you configure the network stack on your OS to perform bandwidth throttling.

For example, the following Linux commands cap the connection between Filebeat and Logstash by setting a limit of 50 kbps on TCP connections over port 5044:

tc qdisc add dev $DEV root handle 1: htb
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 50kbps ceil 50kbps
tc filter add dev $DEV parent 1:0 prio 1 protocol ip handle 10 fw flowid 1:10
iptables -A OUTPUT -t mangle -p tcp --dport 5044 -j MARK --set-mark 10

Using OS tools to perform bandwidth throttling gives you better control over policies. For example, you can use OS tools to cap bandwidth during the day, but not at night. Or you can leave the bandwidth uncapped, but assign a low priority to the traffic.

You may have configured Logstash or Filebeat incorrectly. To resolve the issue:

Logstash outputs remove @metadata fields automatically. Therefore, if Logstash instances are chained directly or via some message queue (for example, Redis or Kafka), the @metadata field will not be available in the final Logstash instance.

Beats are lightweight data shippers that you install as agents on your servers to send specific types of operational data to Elasticsearch. Beats have a small footprint and use fewer system resources than Logstash.

Logstash has a larger footprint, but provides a broad array of input, filter, and output plugins for collecting, enriching, and transforming data from a variety of sources.

For more information, see the Logstash Introduction and the Beats Overview.

The host running Logstash might be unreachable or the certificate may not be valid. To resolve your issue:

Here are some common errors and ways to fix them:

This happens because your certificate is only valid for the hostname present in the Subject field.

To resolve this problem, try one of these solutions:

This is not a TLS problem. It’s a networking problem. Make sure the two hosts can communicate.

This is not a TLS problem. Make sure that Logstash is running and that there is no firewall blocking the traffic.

A firewall is refusing the connection. Check if a firewall is blocking the traffic on the client, the network, or the destination host.