Secure your connection to Elasticsearch
editSecure your connection to Elasticsearch
editThe Logstash Elasticsearch output, input, and filter plugins, as well as monitoring and central management, support authentication and encryption over HTTPS.
Elasticsearch clusters are secured by default (starting in 8.0). You need to configure authentication credentials for Logstash in order to establish communication. Logstash throws an exception and the processing pipeline is halted if authentication fails.
In addition to configuring authentication credentials for Logstash, you need to grant authorized users permission to access the Logstash indices.
Security is enabled by default on the Elasticsearch cluster (starting in 8.0). You must enable TLS/SSL in the Elasticsearch output section of the Logstash configuration in order to allow Logstash to communicate with the Elasticsearch cluster.
Elasticsearch security on by default
editElasticsearch generates its own default self-signed Secure Sockets Layer (SSL) certificates at startup.
Logstash must establish a Secure Sockets Layer (SSL) connection before it can transfer data to a secured Elasticsearch cluster. Logstash must have a copy of the certificate authority (CA) that signed the Elasticsearch cluster’s certificates. When a new Elasticsearch cluster is started up without dedicated certificates, it generates its own default self-signed Certificate Authority at startup. See Starting the Elastic Stack with security enabled for more info.
Elasticsearch Service uses certificates signed by standard publicly trusted certificate authorities, and therefore setting a cacert is not necessary.
Hosted Elasticsearch Service simplifies security
Our hosted Elasticsearch Service on Elastic Cloud simplifies safe, secure communication between Logstash and Elasticsearch.
When you configure the elasticsearch output plugin to use cloud_id
with either the cloud_auth
option or the api_key
option, no additional SSL configuration steps are needed.
Examples:
-
output {elasticsearch { cloud_id => "<cloud id>" cloud_auth => "<cloud auth>" } }
-
output {elasticsearch { cloud_id => "<cloud id>" api_key => "<api key>" } }`
For more details, check out the Logstash-to-Cloud documentation.
Our hosted Elasticsearch Service is available on AWS, GCP, and Azure, and you can try it for free.
Secure communication with an on-premise Elasticsearch cluster
editIf you are running Elasticsearch on your own hardware and using the Elasticsearch cluster’s default self-signed certificates, you need to complete a few more steps to establish secure communication between Logstash and Elasticsearch.
You need to:
- Copy the self-signed CA certificate from Elasticsearch and save it to Logstash.
- Configure the elasticsearch-output plugin to use the certificate.
These steps are not necessary if your cluster is using public trusted certificates.
Copy and save the certificate
editBy default an on-premise Elasticsearch cluster generates a self-signed CA and creates its own SSL certificates when it starts. Therefore Logstash needs its own copy of the self-signed CA from the Elasticsearch cluster in order for Logstash to validate the certificate presented by Elasticsearch.
Copy the self-signed CA certificate from the Elasticsearch config/certs
directory.
Save it to a location that Logstash can access, such as config/certs
on the Logstash instance.
Configure the elasticsearch output
editUse the elasticsearch output
's cacert
option to point to the certificate’s location.
Example
Note that the |
|
Path to the Logstash copy of the Elasticsearch certificate |
For more information about establishing secure communication with Elasticsearch, see security is on by default.
Configuring Logstash to use basic authentication
editLogstash needs to be able to manage index templates, create indices, and write and delete documents in the indices it creates.
To set up authentication credentials for Logstash:
-
Use the the Management > Roles UI in Kibana or the
role
API to create alogstash_writer
role. For cluster privileges, addmanage_index_templates
andmonitor
. For indices privileges, addwrite
,create
, andcreate_index
.Add
manage_ilm
for cluster andmanage
andmanage_ilm
for indices if you plan to use index lifecycle management.POST _security/role/logstash_writer { "cluster": ["manage_index_templates", "monitor", "manage_ilm"], "indices": [ { "names": [ "logstash-*" ], "privileges": ["write","create","create_index","manage","manage_ilm"] } ] }
The cluster needs the
manage_ilm
privilege if index lifecycle management is enabled.If you use a custom Logstash index pattern, specify your custom pattern instead of the default
logstash-*
pattern.If index lifecycle management is enabled, the role requires the
manage
andmanage_ilm
privileges to load index lifecycle policies, create rollover aliases, and create and manage rollover indices. -
Create a
logstash_internal
user and assign it thelogstash_writer
role. You can create users from the Management > Users UI in Kibana or through theuser
API:POST _security/user/logstash_internal { "password" : "x-pack-test-password", "roles" : [ "logstash_writer"], "full_name" : "Internal Logstash User" }
-
Configure Logstash to authenticate as the
logstash_internal
user you just created. You configure credentials separately for each of the Elasticsearch plugins in your Logstash.conf
file. For example:input { elasticsearch { ... user => logstash_internal password => x-pack-test-password } } filter { elasticsearch { ... user => logstash_internal password => x-pack-test-password } } output { elasticsearch { ... user => logstash_internal password => x-pack-test-password } }
Granting access to the Logstash indices
editTo access the indices Logstash creates, users need the read
and
view_index_metadata
privileges:
-
Create a
logstash_reader
role that has theread
andview_index_metadata
privileges for the Logstash indices. You can create roles from the Management > Roles UI in Kibana or through therole
API:POST _security/role/logstash_reader { "cluster": ["manage_logstash_pipelines"] }
-
Assign your Logstash users the
logstash_reader
role. If the Logstash user will be using centralized pipeline management, also assign thelogstash_admin
role. You can create and manage users from the Management > Users UI in Kibana or through theuser
API:
Configuring Logstash to use TLS/SSL encryption
editIf TLS encryption is enabled on an on premise Elasticsearch cluster, you need to
configure the ssl
and cacert
options in your Logstash .conf
file:
Hosted Elasticsearch Service simplifies security. This configuration step is not necessary for hosted Elasticsearch Service on Elastic Cloud. Our hosted Elasticsearch Service is available on AWS, GCP, and Azure, and you can try it for free.
Configuring the Elasticsearch output to use PKI authentication
editThe elasticsearch
output supports PKI authentication. To use an X.509
client-certificate for authentication, you configure the keystore
and
keystore_password
options in your Logstash .conf
file:
output { elasticsearch { ... keystore => /path/to/keystore.jks keystore_password => realpassword truststore => /path/to/truststore.jks truststore_password => realpassword } }
Configuring credentials for Logstash monitoring
editIf you want to monitor your Logstash instance with Elastic Stack monitoring features, and store the monitoring data in a secured Elasticsearch cluster, you must configure Logstash with a username and password for a user with the appropriate permissions.
The security features come preconfigured with a
logstash_system
built-in user
for this purpose. This user has the minimum permissions necessary for the
monitoring function, and should not be used for any other purpose - it is
specifically not intended for use within a Logstash pipeline.
By default, the logstash_system
user does not have a password. The user will
not be enabled until you set a password. See
Setting built-in user passwords.
Then configure the user and password in the logstash.yml
configuration file:
xpack.monitoring.elasticsearch.username: logstash_system xpack.monitoring.elasticsearch.password: t0p.s3cr3t
If you initially installed an older version of X-Pack and then upgraded, the
logstash_system
user may have defaulted to disabled
for security reasons.
You can enable the user through the user
API:
PUT _security/user/logstash_system/_enable
Configuring credentials for Centralized Pipeline Management
editIf you plan to use Logstash centralized pipeline management, you need to configure the username and password that Logstash uses for managing configurations.
You configure the user and password in the logstash.yml
configuration file:
xpack.management.elasticsearch.username: logstash_admin_user xpack.management.elasticsearch.password: t0p.s3cr3t
The user you specify here must have the built-in |
Grant access using API keys
editInstead of using usernames and passwords, you can use API keys to grant
access to Elasticsearch resources. You can set API keys to expire at a certain time,
and you can explicitly invalidate them. Any user with the manage_api_key
or manage_own_api_key
cluster privilege can create API keys.
Tips for creating API keys:
- API keys are tied to the cluster they are created in. If you are sending output to different clusters, be sure to create the correct kind of API key.
- Logstash can send both collected data and monitoring information to Elasticsearch. If you are sending both to the same cluster, you can use the same API key. For different clusters, you need an API key per cluster.
- A single cluster can share a key for ingestion and monitoring purposes.
- A production cluster and a monitoring cluster require separate keys.
For security reasons, we recommend using a unique API key per Logstash instance. You can create as many API keys per user as necessary.
Create an API key
editYou can create API keys using either the Create API key API or the Kibana UI. This section walks you through creating an API key using the Create API key API. The privileges needed are the same for either approach.
Here is an example that shows how to create an API key for publishing to Elasticsearch using the Elasticsearch output plugin.
POST /_security/api_key { "name": "logstash_host001", "role_descriptors": { "logstash_writer": { "cluster": ["monitor", "manage_ilm", "read_ilm"], "index": [ { "names": ["logstash-*"], "privileges": ["view_index_metadata", "create_doc"] } ] } } }
The return value should look similar to this:
Create an API key for publishing
editYou’re in luck! The example we used in the Create an API key section creates an API key for publishing to Elasticsearch using the Elasticsearch output plugin.
Here’s an example using the API key in your Elasticsearch output plugin configuration.
Format is |
Create an API key for reading
editCreating an API key to use for reading data from Elasticsearch is similar to creating an API key for publishing described earlier. You can use the example in the Create an API key section, granting the appropriate privileges.
Here’s an example using the API key in your Elasticsearch inputs plugin configuration.
Format is |
Create an API key for filtering
editCreating an API key to use for processing data from Elasticsearch is similar to creating an API key for publishing described earlier. You can use the example in the Create an API key section, granting the appropriate privileges.
Here’s an example using the API key in your Elasticsearch filter plugin configuration.
Format is |
Create an API key for monitoring
editTo create an API key to use for sending monitoring data to Elasticsearch, use the Create API key API. For example:
POST /_security/api_key { "name": "logstash_host001", "role_descriptors": { "logstash_monitoring": { "cluster": ["monitor"], "index": [ { "names": [".monitoring-ls-*"], "privileges": ["create_index", "create"] } ] } } }
The return value should look similar to this:
Now you can use this API key in your logstash.yml configuration file:
Format is |
Create an API key for central management
editTo create an API key to use for central management, use the Create API key API. For example:
POST /_security/api_key { "name": "logstash_host001", "role_descriptors": { "logstash_monitoring": { "cluster": ["monitor"], "index": ["read"] } } }
The return value should look similar to this:
Now you can use this API key in your logstash.yml configuration file:
Format is |
Learn more about API keys
editSee the Elasticsearch API key documentation for more information:
See API Keys for info on managing API keys through Kibana.