s3

This plugin will recover and upload temporary log files after crash/abnormal termination

Both time_file and size_file settings can trigger a log "file rotation" A log rotation pushes the current log "part" to s3 and deleted from local temporary storage.

This is an example of logstash config:

output {
   s3{
     access_key_id => "crazy_key"             (required)
     secret_access_key => "monkey_access_key" (required)
     region => "eu-west-1"                    (optional, default = "us-east-1")
     bucket => "boss_please_open_your_bucket" (required)
     size_file => 2048                        (optional) - Bytes
     time_file => 5                           (optional) - Minutes
     format => "plain"                        (optional)
     canned_acl => "private"                  (optional. Options are "private", "public_read", "public_read_write", "authenticated_read", "bucket_owner_full_control". Defaults to "private" )
   }

This plugin uses the AWS SDK and supports several ways to get credentials, which will be tried in this order:

Path to YAML file containing a hash of AWS credentials. This file will only be loaded if access_key_id and secret_access_key aren’t set. The contents of the file should look like this:

    :access_key_id: "12345"
    :secret_access_key: "54321"

S3 bucket

The S3 canned ACL to use when putting the file. Defaults to "private".

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

Disable or enable metric logging for this specific plugin instance by default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

Specify the content encoding. Supports ("gzip"). Defaults to "none"

Add a unique ID to the plugin instance, this ID is used for tracking information for a specific configuration of the plugin.

output {
 stdout {
   id => "ABC"
 }
}

If you don’t explicitely set this variable Logstash will generate a unique name.

Specify a prefix to the uploaded filename, this can simulate directories on S3. Prefix does not require leading slash.

URI to proxy server if required

The AWS Region

The AWS Secret Access Key

Specifies wether or not to use S3’s AES256 server side encryption. Defaults to false.

The AWS Session token for temporary credential

The version of the S3 signature hash to use. Normally uses the internal client default, can be explicitly specified here

Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file. If you have tags then it will generate a specific size file for every tags

Define tags to be appended to the file on the S3 bucket.

Example: tags ⇒ ["elasticsearch", "logstash", "kibana"]

Will generate this file: "ls.s3.logstash.local.2015-01-01T00.00.tag_elasticsearch.logstash.kibana.part0.txt"

Set the directory where logstash will store the tmp files before sending it to S3 default to the current OS temporary directory in linux /tmp/logstash

Set the time, in MINUTES, to close the current sub_time_section of bucket. If you define file_size you have a number of files in consideration of the section and the current tag. 0 stay all time on listerner, beware if you specific 0 and size_file 0, because you will not put the file on bucket, for now the only thing this plugin can do is to put the file when logstash restart.

Specify how many workers to use to upload the files to S3

Should we require (true) or disable (false) using SSL for communicating with the AWS API The AWS SDK for Ruby defaults to SSL so we preserve that

TODO remove this in Logstash 6.0 when we no longer support the :legacy type This is hacky, but it can only be herne