IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« udp websocket »

› ›

webhdfs

edit

webhdfs

edit

Version: 3.0.2
Released on: 2016-07-14
Changelog
Compatible: 5.1.1.1, 5.0.0, 2.4.1, 2.4.0, 2.3.4

This plugin sends Logstash events into files in HDFS via the webhdfs REST API.

Dependencies

edit

This plugin has no dependency on jars from hadoop, thus reducing configuration and compatibility problems. It uses the webhdfs gem from Kazuki Ohta and TAGOMORI Satoshi (@see: https://github.com/kzk/webhdfs). Optional dependencies are zlib and snappy gem if you use the compression functionality.

Operational Notes

edit

If you get an error like:

Max write retries reached. Exception: initialize: name or service not known {:level=>:error}

make sure that the hostname of your namenode is resolvable on the host running Logstash. When creating/appending to a file, webhdfs somtime sends a 307 TEMPORARY_REDIRECT with the HOSTNAME of the machine its running on.

Usage

edit

This is an example of Logstash config:

input {
  ...
}
filter {
  ...
}
output {
  webhdfs {
    host => "127.0.0.1"                 # (required)
    port => 50070                       # (optional, default: 50070)
    path => "/user/logstash/dt=%{+YYYY-MM-dd}/logstash-%{+HH}.log"  # (required)
    user => "hue"                       # (required)
  }
}

Synopsis

edit

This plugin supports the following configuration options:

Required configuration options:

webhdfs {
    host => ...
    path => ...
    user => ...
}

Available configuration options:

Setting	Input type	Required	Default value
`codec`	codec	No	`"plain"`
`compression`	string, one of `["none", "snappy", "gzip"]`	No	`"none"`
`enable_metric`	boolean	No	`true`
`flush_size`	number	No	`500`
`host`	string	Yes
`id`	string	No
`idle_flush_time`	number	No	`1`
`open_timeout`	number	No	`30`
`path`	string	Yes
`port`	number	No	`50070`
`read_timeout`	number	No	`30`
`retry_interval`	number	No	`0.5`
`retry_known_errors`	boolean	No	`true`
`retry_times`	number	No	`5`
`single_file_per_thread`	boolean	No	`false`
`snappy_bufsize`	number	No	`32768`
`snappy_format`	string, one of `["stream", "file"]`	No	`"stream"`
`standby_host`	string	No	`false`
`standby_port`	number	No	`50070`
`use_httpfs`	boolean	No	`false`
`user`	string	Yes
`workers`	<<,>>	No	`1`

Details

edit

`codec`

edit

Value type is codec
Default value is "plain"

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

`compression`

edit

Value can be any of: none, snappy, gzip
Default value is "none"

Compress output. One of [none, snappy, gzip]

`enable_metric`

edit

Value type is boolean
Default value is true

Disable or enable metric logging for this specific plugin instance by default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

`flush_size`

edit

Value type is number
Default value is 500

Sending data to webhdfs if event count is above, even if store_interval_in_secs is not reached.

`host`

edit

This is a required setting.
Value type is string
There is no default value for this setting.

The server name for webhdfs/httpfs connections.

`id`

edit

Value type is string
There is no default value for this setting.

Add a unique ID to the plugin instance, this ID is used for tracking information for a specific configuration of the plugin.

output {
 stdout {
   id => "ABC"
 }
}

If you don’t explicitely set this variable Logstash will generate a unique name.

`idle_flush_time`

edit

Value type is number
Default value is 1

Sending data to webhdfs in x seconds intervals.

`open_timeout`

edit

Value type is number
Default value is 30

WebHdfs open timeout, default 30s.

`path`

edit

This is a required setting.
Value type is string
There is no default value for this setting.

The path to the file to write to. Event fields can be used here, as well as date fields in the joda time format, e.g.: /user/logstash/dt=%{+YYYY-MM-dd}/%{@source_host}-%{+HH}.log

`port`

edit

Value type is number
Default value is 50070

The server port for webhdfs/httpfs connections.

`read_timeout`

edit

Value type is number
Default value is 30

The WebHdfs read timeout, default 30s.

`retry_interval`

edit

Value type is number
Default value is 0.5

How long should we wait between retries.

`retry_known_errors`

edit

Value type is boolean
Default value is true

Retry some known webhdfs errors. These may be caused by race conditions when appending to same file, etc.

`retry_times`

edit

Value type is number
Default value is 5

How many times should we retry. If retry_times is exceeded, an error will be logged and the event will be discarded.

`single_file_per_thread`

edit

Value type is boolean
Default value is false

Avoid appending to same file in multiple threads. This solves some problems with multiple logstash output threads and locked file leases in webhdfs. If this option is set to true, %{[@metadata][thread_id]} needs to be used in path config settting.