Sending data to Elasticsearch Serverless
When you use Elasticsearch on Elastic Cloud Serverless you don’t need to worry about managing the infrastructure that keeps Elasticsearch distributed and available. These resources are automated on the serverless platform and are designed to scale up and down with your workload.
You’ll use the Logstash Elasticsearch output plugin to send data to Elastic Cloud Serverless. Note these differences between Elasticsearch Serverless and both Elastic Cloud Hosted and self-managed Elasticsearch:
- Use API keys to access Elastic Cloud Serverless from Logstash as it does not support native user authentication. Any user-based security settings in your Elasticsearch output plugin configuration are ignored and may cause errors.
- Elastic Cloud Serverless uses data streams and data lifecycle management (DLM) instead of index lifecycle management (ILM). Any ILM settings in your Elasticsearch output plugin configuration are ignored and may cause errors.
- Logstash monitoring is available through the Logstash Integration in Elastic Observability on Elastic Cloud Serverless.
Known issue for Logstash to Elasticsearch Serverless.
The logstash-output-elasticsearch hosts setting defaults to port :9200.
Set the value to port :443 instead.
Elasticsearch Serverless simplifies safe, secure communication between Logstash and Elasticsearch. To send data to a Serverless project, configure the Logstash Elasticsearch output plugin to connect using the project's Elasticsearch endpoint URL and an API key.
output {elasticsearch { hosts => "ELASTICSEARCH_ENDPOINT_URL" api_key => "<api key>" } }
The value of the api_key option is in the format id:api_key, where id and api_key are the values returned by the Create API key API.
Log in to Elastic Cloud.
Find your Elasticsearch endpoint URL:
Select Manage next to your project. Then find the Elasticsearch endpoint under Application endpoints, cluster and component IDs.
Alternatively, open your project, select the help icon, then select Connection details.
Create an API key with the appropriate privileges. Refer to Create API key for detailed steps. For information on the required privileges, refer to Grant access using API keys.
When you create an API key for Logstash, select Logstash from the API key format dropdown.
This option formats the API key in the correct id:api_key format required by Logstash.
The Elasticsearch input,
Elasticsearch output, and
Elasticsearch filter plugins, as well as the
Elastic_integration filter plugin, all support the api_key option in their configurations.
To set up Central Pipeline management in Elasticsearch Serverless, update the logstash.yml config file to provide the API key and set the value for xpack.management.elasticsearch.hosts to your Elasticsearch endpoint URL.
xpack.management.elasticsearch.hosts: "ELASTICSEARCH_ENDPOINT_URL"
xpack.management.elasticsearch.api_key: "<api_key>"
- Use the `hosts` option with the Elasticsearch endpoint URL to connect to Elasticsearch Serverless
- API key formatted for Logstash: `id:api_key`