IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« riak sqs »

› › ›

s3

edit

s3

edit

INFORMATION:

This plugin was created for store the logstash’s events into Amazon Simple Storage Service (Amazon S3). For use it you needs authentications and an s3 bucket. Be careful to have the permission to write file on S3’s bucket and run logstash with super user for establish connection.

S3 plugin allows you to do something complex, let’s explain:)

S3 outputs create temporary files into "/opt/logstash/S3_temp/". If you want, you can change the path at the start of register method. This files have a special name, for example:

ls.s3.ip-10-228-27-95.2013-04-18T10.00.tag_hello.part0.txt

ls.s3 : indicate logstash plugin s3

"ip-10-228-27-95" : indicate you ip machine, if you have more logstash and writing on the same bucket for example. "2013-04-18T10.00" : represents the time whenever you specify time_file. "tag_hello" : this indicate the event’s tag, you can collect events with the same tag. "part0" : this means if you indicate size_file then it will generate more parts if you file.size > size_file. When a file is full it will pushed on bucket and will be deleted in temporary directory. If a file is empty is not pushed, but deleted.

This plugin have a system to restore the previous temporary files if something crash.

Usage:

edit

This is an example of logstash config:

output {
   s3{
     access_key_id => "crazy_key"             (required)
     secret_access_key => "monkey_access_key" (required)
     endpoint_region => "eu-west-1"           (required)
     bucket => "boss_please_open_your_bucket" (required)
     size_file => 2048                        (optional)
     time_file => 5                           (optional)
     format => "plain"                        (optional)
     canned_acl => "private"                  (optional. Options are "private", "public_read", "public_read_write", "authenticated_read". Defaults to "private" )
   }

Synopsis

edit

This plugin supports the following configuration options:

Required configuration options:

s3 {
}

Available configuration options:

Setting	Input type	Required	Default value
`access_key_id`	string	No
`aws_credentials_file`	string	No
`bucket`	string	No
`canned_acl`	string, one of `["private", "public_read", "public_read_write", "authenticated_read"]`	No	`"private"`
`codec`	codec	No	`"line"`
`prefix`	string	No	`""`
`proxy_uri`	string	No
`region`	string, one of `["us-east-1", "us-west-1", "us-west-2", "eu-central-1", "eu-west-1", "ap-southeast-1", "ap-southeast-2", "ap-northeast-1", "sa-east-1", "us-gov-west-1", "cn-north-1"]`	No	`"us-east-1"`
`restore`	boolean	No	`false`
`secret_access_key`	string	No
`session_token`	string	No
`size_file`	number	No	`0`
`temporary_directory`	string	No	`"/var/folders/_9/x4bq65rs6vd0rrjthct3zxjw0000gn/T/logstash"`
`time_file`	number	No	`0`
`upload_workers_count`	number	No	`1`
`use_ssl`	boolean	No	`true`
`workers`	number	No	`1`

Details

edit

`access_key_id`

edit

Value type is string
There is no default value for this setting.

`aws_credentials_file`

edit

Value type is string
There is no default value for this setting.

`bucket`

edit

Value type is string
There is no default value for this setting.

S3 bucket

`canned_acl`

edit

Value can be any of: private, public_read, public_read_write, authenticated_read
Default value is "private"

The S3 canned ACL to use when putting the file. Defaults to "private".

`codec`

edit

Value type is codec
Default value is "line"

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

`endpoint_region` (DEPRECATED)

edit

DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
Value can be any of: us-east-1, us-west-1, us-west-2, eu-west-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1, us-gov-west-1
There is no default value for this setting.

AWS endpoint_region

`exclude_tags` (DEPRECATED)

edit

DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
Value type is array
Default value is []

Only handle events without any of these tags. Optional.

`prefix`

edit

Value type is string
Default value is ""

Specify a prefix to the uploaded filename, this can simulate directories on S3

`proxy_uri`

edit

Value type is string
There is no default value for this setting.

`region`

edit

Value can be any of: us-east-1, us-west-1, us-west-2, eu-central-1, eu-west-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, sa-east-1, us-gov-west-1, cn-north-1
Default value is "us-east-1"

`restore`

edit

Value type is boolean
Default value is false

`secret_access_key`

edit

Value type is string
There is no default value for this setting.

`session_token`

edit

Value type is string
There is no default value for this setting.

`size_file`

edit

Value type is number
Default value is 0

Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file. If you have tags then it will generate a specific size file for every tags

`tags` (DEPRECATED)

edit

DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
Value type is array
Default value is []

Only handle events with all of these tags. Optional.

`temporary_directory`

edit

Value type is string
Default value is "/var/folders/_9/x4bq65rs6vd0rrjthct3zxjw0000gn/T/logstash"

Set the directory where logstash will store the tmp files before sending it to S3 default to the current OS temporary directory in linux /tmp/logstash

`time_file`

edit

Value type is number
Default value is 0

Set the time, in minutes, to close the current sub_time_section of bucket. If you define file_size you have a number of files in consideration of the section and the current tag. 0 stay all time on listerner, beware if you specific 0 and size_file 0, because you will not put the file on bucket, for now the only thing this plugin can do is to put the file when logstash restart.

`type` (DEPRECATED)

edit

DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
Value type is string
Default value is ""

The type to act on. If a type is given, then this output will only act on messages with the same type. See any input plugin’s type attribute for more. Optional.

`upload_workers_count`

edit

Value type is number
Default value is 1

Specify how many workers to use to upload the files to S3

`use_ssl`

edit

Value type is boolean
Default value is true

`workers`

edit

Value type is number
Default value is 1

The number of workers to use for this output. Note that this setting may not be useful for all outputs.

« riak sqs »

s3

s3

Usage:

Synopsis

Details

access_key_id

aws_credentials_file

bucket

canned_acl

codec

endpoint_region (DEPRECATED)

exclude_tags (DEPRECATED)

prefix

proxy_uri

region

restore

secret_access_key

session_token

size_file

tags (DEPRECATED)

temporary_directory

time_file

type (DEPRECATED)

upload_workers_count

use_ssl

workers

`access_key_id`

`aws_credentials_file`

`bucket`

`canned_acl`

`codec`

`endpoint_region` (DEPRECATED)

`exclude_tags` (DEPRECATED)

`prefix`

`proxy_uri`

`region`

`restore`

`secret_access_key`

`session_token`

`size_file`

`tags` (DEPRECATED)

`temporary_directory`

`time_file`

`type` (DEPRECATED)

`upload_workers_count`

`use_ssl`

`workers`