S3 output plugin

This plugin will recover and upload temporary log files after crash/abnormal termination when using restore set to true

This is an example of logstash config:

output {
   s3{
     access_key_id => "crazy_key"             (optional)
     secret_access_key => "monkey_access_key" (optional)
     region => "eu-west-1"                    (optional, default = "us-east-1")
     bucket => "your_bucket"                  (required)
     size_file => 2048                        (optional) - Bytes
     time_file => 5                           (optional) - Minutes
     codec => "plain"                         (optional)
     canned_acl => "private"                  (optional. Options are "private", "public-read", "public-read-write", "authenticated-read", "aws-exec-read", "bucket-owner-read", "bucket-owner-full-control", "log-delivery-write". Defaults to "private" )
   }

This plugin uses the AWS SDK and supports several ways to get credentials, which will be tried in this order:

Key-value pairs of settings and corresponding values used to parametrize the connection to S3. See full list in the AWS SDK documentation. Example:

    output {
      s3 {
        access_key_id => "1234",
        secret_access_key => "secret",
        region => "eu-west-1",
        bucket => "logstash-test",
        additional_settings => {
          "force_path_style" => true,
          "follow_redirects" => false
        }
      }
    }

Path to YAML file containing a hash of AWS credentials. This file will only be loaded if access_key_id and secret_access_key aren’t set. The contents of the file should look like this:

    :access_key_id: "12345"
    :secret_access_key: "54321"

S3 bucket

The S3 canned ACL to use when putting the file. Defaults to "private".

Specify the content encoding. Supports ("gzip"). Defaults to "none"

The endpoint to connect to. By default it is constructed using the value of region. This is useful when connecting to S3 compatible services, but beware that these aren’t guaranteed to work correctly with the AWS SDK. The endpoint should be an HTTP or HTTPS URL, e.g. https://example.com

Specify a prefix to the uploaded filename, this can simulate directories on S3. Prefix does not require leading slash. This option supports logstash interpolation: https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#sprintf; for example, files can be prefixed with the event date using prefix = "%{+YYYY}/%{+MM}/%{+dd}". Be warned this can create a lot of temporary local files.

URI to proxy server if required

The AWS Region

Used to enable recovery after crash/abnormal termination. Temporary log files will be recovered and uploaded.

The AWS IAM Role to assume, if any. This is used to generate temporary credentials, typically for cross-account access. See the AssumeRole API documentation for more information.

Session name to use when assuming an IAM role.

Define the strategy to use to decide when we need to rotate the file and push it to S3, The default strategy is to check for both size and time, the first one to match will rotate the file.

The AWS Secret Access Key

Specifies whether or not to use S3’s server side encryption. Defaults to no encryption.

Specifies what type of encryption to use when SSE is enabled.

The AWS Session token for temporary credential

The version of the S3 signature hash to use. Normally uses the internal client default, can be explicitly specified here

Set the size of file in bytes, this means that files on bucket when have dimension > file_size, they are stored in two or more file. If you have tags then it will generate a specific size file for every tags

The key to use when specified along with server_side_encryption ⇒ aws:kms. If server_side_encryption ⇒ aws:kms is set but this is not default KMS key is used. http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html

Specifies what S3 storage class to use when uploading the file. More information about the different storage classes can be found: http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html Defaults to STANDARD.

Set the directory where logstash will store the tmp files before sending it to S3 default to the current OS temporary directory in linux /tmp/logstash

Set the time, in MINUTES, to close the current sub_time_section of bucket. If you define file_size you have a number of files in consideration of the section and the current tag. 0 stay all time on listener, beware if you specific 0 and size_file 0, because you will not put the file on bucket, for now the only thing this plugin can do is to put the file when logstash restart.

Files larger than this number are uploaded using the S3 multipart APIs

Number of items we can keep in the local queue before uploading them

Specify how many workers to use to upload the files to S3

The common use case is to define permission on the root bucket and give Logstash full access to write its logs. In some circumstances you need finer grained permission on subfolder, this allow you to disable the check at startup.

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output without needing a separate filter in your Logstash pipeline.

Disable or enable metric logging for this specific plugin instance. By default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

Add a unique ID to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type. For example, if you have 2 s3 outputs. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.

output {
  s3 {
    id => "my_plugin_id"
  }
}