Configuring Security in Logstash

The Logstash Elasticsearch plugins (output, input, filter and monitoring) support authentication and encryption over HTTPS.

To use Logstash with a secured cluster, you need to configure authentication credentials for Logstash. Logstash throws an exception and the processing pipeline is halted if authentication fails.

If encryption is enabled on the cluster, you also need to enable TLS/SSL in the Logstash configuration.

If you want to monitor your Logstash instance with X-Pack monitoring, and store the monitoring data in a secured Elasticsearch cluster, you must configure Logstash with a username and password for a user with the appropriate permissions.

In addition to configuring authentication credentials for Logstash, you need to grant authorized users permission to access the Logstash indices.

Logstash needs to be able to manage index templates, create indices, and write and delete documents in the indices it creates.

To set up authentication credentials for Logstash:

To access the indices Logstash creates, users need the read and view_index_metadata privileges:

The elasticsearch output supports PKI authentication. To use an X.509 client-certificate for authentication, you configure the keystore and keystore_password options in your Logstash .conf file:

output {
  elasticsearch {
    ...
    keystore => /path/to/keystore.jks
    keystore_password => realpassword
    truststore =>  /path/to/truststore.jks 
    truststore_password =>  realpassword
  }
}

If TLS encryption is enabled on the Elasticsearch cluster, you need to configure the ssl and cacert options in your Logstash .conf file:

output {
  elasticsearch {
    ...
    ssl => true
    cacert => '/path/to/cert.pem' 
  }
}

If you plan to ship Logstash monitoring data to a secure cluster, you need to configure the username and password that Logstash uses to authenticate for shipping monitoring data.

The security features come preconfigured with a logstash_system built-in user for this purpose. This user has the minimum permissions necessary for the monitoring function, and should not be used for any other purpose - it is specifically not intended for use within a Logstash pipeline.

By default, the logstash_system user does not have a password. The user will not be enabled until you set a password. See Setting built-in user passwords.

Then configure the user and password in the logstash.yml configuration file:

xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: t0p.s3cr3t

If you initially installed an older version of X-Pack and then upgraded, the logstash_system user may have defaulted to disabled for security reasons. You can enable the user through the user API:

PUT _xpack/security/user/logstash_system/_enable

If you plan to use Logstash centralized pipeline management, you need to configure the username and password that Logstash uses for managing configurations.

You configure the user and password in the logstash.yml configuration file:

xpack.management.elasticsearch.username: logstash_admin_user 
xpack.management.elasticsearch.password: t0p.s3cr3t

Instead of using usernames and passwords, you can use API keys to grant access to Elasticsearch resources. You can set API keys to expire at a certain time, and you can explicitly invalidate them. Any user with the manage_api_key or manage_own_api_key cluster privilege can create API keys.

Tips for creating API keys:

You can create API keys using either the Create API key API or the Kibana UI. This section walks you through creating an API key using the Create API key API. The privileges needed are the same for either approach.

Here is an example that shows how to create an API key for publishing to Elasticsearch using the Elasticsearch output plugin.

POST /_security/api_key
{
  "name": "logstash_host001", 
  "role_descriptors": {
    "logstash_writer": { 
      "cluster": ["monitor", "manage_ilm", "read_ilm"],
      "index": [
        {
          "names": ["logstash-*"],
          "privileges": ["view_index_metadata", "create_doc"]
        }
      ]
    }
  }
}

The return value should look similar to this:

{
  "id":"TiNAGG4BaaMdaH1tRfuU", 
  "name":"logstash_host001",
  "api_key":"KnR6yE41RrSowb0kQ0HWoA" 
}

You’re in luck! The example we used in the Create an API key section creates an API key for publishing to Elasticsearch using the Elasticsearch output plugin.

Here’s an example using the API key in your Elasticsearch output plugin configuration.

output {
  elasticsearch {
    api_key => "TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSowb0kQ0HWoA" 
  }
}

Creating an API key to use for reading data from Elasticsearch is similar to creating an API key for publishing described earlier. You can use the example in the Create an API key section, granting the appropriate privileges.

Here’s an example using the API key in your Elasticsearch inputs plugin configuration.

input {
  elasticsearch {
    "api_key" => "TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSowb0kQ0HWoA" 
  }
}

Creating an API key to use for processing data from Elasticsearch is similar to creating an API key for publishing described earlier. You can use the example in the Create an API key section, granting the appropriate privileges.

Here’s an example using the API key in your Elasticsearch filter plugin configuration.

filter {
  elasticsearch {
    api_key => "TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSowb0kQ0HWoA" 
  }
}

To create an API key to use for sending monitoring data to Elasticsearch, use the Create API key API. For example:

POST /_security/api_key
{
  "name": "logstash_host001", 
  "role_descriptors": {
    "logstash_monitoring": { 
      "cluster": ["monitor"],
      "index": [
        {
          "names": [".monitoring-ls-*"],
          "privileges": ["create_index", "create"]
        }
      ]
    }
  }
}

The return value should look similar to this:

{
  "id":"TiNAGG4BaaMdaH1tRfuU", 
  "name":"logstash_host001",
  "api_key":"KnR6yE41RrSowb0kQ0HWoA" 
}

Now you can use this API key in your logstash.yml configuration file:

xpack.monitoring.elasticsearch.api_key: TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSowb0kQ0HWoA 

To create an API key to use for central management, use the Create API key API. For example:

POST /_security/api_key
{
  "name": "logstash_host001", 
  "role_descriptors": {
    "logstash_monitoring": { 
      "cluster": ["monitor"],
      "index": ["read"]
    }
  }
}

The return value should look similar to this:

{
  "id":"TiNAGG4BaaMdaH1tRfuU", 
  "name":"logstash_host001",
  "api_key":"KnR6yE41RrSowb0kQ0HWoA" 
}

Now you can use this API key in your logstash.yml configuration file:

xpack.management.elasticsearch.api_key: TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSowb0kQ0HWoA 

See the Elasticsearch API key documentation for more information:

See API Keys for info on managing API keys through Kibana.