File input plugin

In this mode the plugin aims to track changing files and emit new content as it’s appended to each file. In this mode, files are seen as a never ending stream of content and EOF has no special significance. The plugin always assumes that there will be more content. When files are rotated, the smaller or zero size is detected, the current position is reset to zero and streaming continues. A delimiter must be seen before the accumulated characters can be emitted as a line.

In this mode the plugin treats each file as if it is content complete, that is, a finite stream of lines and now EOF is significant. A last delimiter is not needed because EOF means that the accumulated characters can be emitted as a line. Further, EOF here means that the file can be closed and put in the "unwatched" state - this automatically frees up space in the active window. This mode also makes it possible to process compressed files as they are content complete. Read mode also allows for an action to take place after processing the file completely.

In the past attempts to simulate a Read mode while still assuming infinite streams was not ideal and a dedicated Read mode is an improvement.

When set to true, this setting verifies that a compressed file is valid before processing it. There are two passes through the file—​one pass to verify that the file is valid, and another pass to process the file.

Validating a compressed file requires more processing time, but can prevent a corrupt archive from causing looping.

The file input closes any files that were last read the specified duration (seconds if a number is specified) ago. This has different implications depending on if a file is being tailed or read. If tailing, and there is a large time gap in incoming data the file can be closed (allowing other files to be opened) but will be queued for reopening when new data is detected. If reading, the file will be closed after closed_older seconds from when the last bytes were read. This setting is retained for backward compatibility if you upgrade the plugin to 4.1.0+, are reading not tailing and do not switch to using Read mode.

set the new line delimiter, defaults to "\n". Note that when reading compressed files this setting is not used, instead the standard Windows or Unix line endings are used.

How often we expand the filename patterns in the path option to discover new files to watch. This value is a multiple to stat_interval, e.g. if stat_interval is "500 ms" then new files files could be discovered every 15 X 500 milliseconds - 7.5 seconds. In practice, this will be the best case because the time taken to read new content needs to be factored in.

Exclusions (matched against the filename, not full path). Filename patterns are valid here, too. For example, if you have

    path => "/var/log/*"

In Tail mode, you might want to exclude gzipped files:

    exclude => "*.gz"

This option can be used in read mode to enforce closing all watchers when file gets read. Can be used in situation when content of the file is static and won’t change during execution. When set to true it also disables active discovery of the files - only files that were in the directories when process was started will be read. It supports sincedb entries. When file was processed once, then modified - next run will only read newly added entries.

When combined with the file_chunk_size, this option sets how many chunks (bands or stripes) are read from each file before moving to the next active file. For example, a file_chunk_count of 32 and a file_chunk_size 32KB will process the next 1MB from each active file. As the default is very large, the file is effectively read to EOF before moving to the next active file.

File content is read off disk in blocks or chunks and lines are extracted from the chunk. See file_chunk_count to see why and when to change this setting from the default.

When in read mode, what action should be carried out when a file is done with. If delete is specified then the file will be deleted. If log is specified then the full path of the file is logged to the file specified in the file_completed_log_path setting. If log_and_delete is specified then both above actions take place.

Which file should the completely read file paths be appended to. Only specify this path to a file when file_completed_action is log or log_and_delete. IMPORTANT: this file is appended to only - it could become very large. You are responsible for file rotation.

Which attribute of a "watched" file should be used to sort them by. Files can be sorted by modified date or full path alphabetic. Previously the processing order of the discovered and therefore "watched" files was OS dependent.

Select between ascending and descending order when sorting "watched" files. If oldest data first is important then the defaults of last_modified + asc are good. If newest data first is more important then opt for last_modified + desc. If you use special naming conventions for the file full paths then perhaps path + asc will help to control the order of file processing.

When the file input discovers a file that was last modified before the specified duration (seconds if a number is specified), the file is ignored. After it’s discovery, if an ignored file is modified it is no longer ignored and any new data is read. By default, this option is disabled. Note this unit is in seconds.

What is the maximum number of file_handles that this input consumes at any one time. Use close_older to close some files if you need to process more files than this number. This should not be set to the maximum the OS can do because file handles are needed for other LS plugins and OS processes. A default of 4095 is set in internally.

What mode do you want the file input to operate in. Tail a few files or read many content-complete files. Read mode now supports gzip file processing.

If read is specified, these settings can be used:

If read is specified, these settings are ignored:

The path(s) to the file(s) to use as an input. You can use filename patterns here, such as /var/log/*.log. If you use a pattern like /var/log/**/*.log, a recursive search of /var/log will be done for all *.log files. Paths must be absolute and cannot be relative.

You may also configure multiple paths. See an example on the Logstash configuration page.

The sincedb record now has a last active timestamp associated with it. If no changes are detected in a tracked file in the last N days its sincedb tracking record expires and will not be persisted. This option helps protect against the inode recycling problem. Filebeat has a FAQ about inode recycling.

Path of the sincedb database file (keeps track of the current position of monitored log files) that will be written to disk. The default will write sincedb files to <path.data>/plugins/inputs/file NOTE: it must be a file path and not a directory path

How often (in seconds) to write a since database with the current position of monitored log files.

Choose where Logstash starts initially reading files: at the beginning or at the end. The default behavior treats files like live streams and thus starts at the end. If you have old data you want to import, set this to beginning.

This option only modifies "first contact" situations where a file is new and not seen before, i.e. files that don’t have a current position recorded in a sincedb file read by Logstash. If a file has already been seen before, this option has no effect and the position recorded in the sincedb file will be used.

How often (in seconds) we stat files to see if they have been modified. Increasing this interval will decrease the number of system calls we make, but increase the time to detect new log lines.

Add a field to an event

The codec used for input data. Input codecs are a convenient method for decoding your data before it enters the input, without needing a separate filter in your Logstash pipeline.

Disable or enable metric logging for this specific plugin instance by default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

Add a unique ID to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type, for example, if you have 2 file inputs. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.

input {
  file {
    id => "my_plugin_id"
  }
}

Add any number of arbitrary tags to your event.

This can help with processing later.

Add a type field to all events handled by this input.

Types are used mainly for filter activation.

The type is stored as part of the event itself, so you can also use the type to search for it in Kibana.

If you try to set a type on an event that already has one (for example when you send an event from a shipper to an indexer) then a new input will not override the existing type. A type set at the shipper stays with that event for its life even when sent to another Logstash server.

Supported values: w week weeks, e.g. "2 w", "1 week", "4 weeks".

Supported values: d day days, e.g. "2 d", "1 day", "2.5 days".

Supported values: h hour hours, e.g. "4 h", "1 hour", "0.5 hours".

Supported values: m min minute minutes, e.g. "45 m", "35 min", "1 minute", "6 minutes".

Supported values: s sec second seconds, e.g. "45 s", "15 sec", "1 second", "2.5 seconds".

Supported values: ms msec msecs, e.g. "500 ms", "750 msec", "50 msecs

Supported values: us usec usecs, e.g. "600 us", "800 usec", "900 usecs"