Logstash
editLogstash
edit[preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
Logstash is an open source data collection engine with real-time pipelining capabilities. It supports a wide variety of data sources, and can dynamically unify data from disparate sources and normalize the data into destinations of your choice.
Logstash can collect data using a variety of Logstash input plugins, enrich and transform the data with Logstash filter plugins, and output the data to Elasticsearch using the Logstash Elasticsearch output plugin.
You can use Logstash to extend Beats for advanced use cases, such as data routed to multiple destinations or when you need to make your data persistent.
Logstash for Elasticsearch on serverless
Logstash is a powerful, versatile ETL (Extract, Transform, Load) engine that can play an important role in organizations of all sizes. Some capabilities and features for large, self-managed users aren’t appropriate for Serverless.
You’ll use the Logstash ((es)) output plugin
to send data to Elasticsearch Serverless.
Some differences to note between Elasticsearch Serverless and self-managed Elasticsearch:
- Your logstash-output-elasticsearch configuration uses API keys to access Elasticsearch from Logstash. User-based security settings are ignored and may cause errors.
- Elasticsearch Serverless uses data lifecycle management (DLM) instead of index lifecycle management (ILM). If you add ILM settings to your Elasticsearch output configuration, they are ignored and may cause errors.
- Logstash monitoring for Serverless is available through the Logstash Integration in Elastic Observability.
Known issue
-
The logstash-output-elasticsearch
hosts
setting defaults to port:9200
. Set the value to port:443
instead.
Requirements
editTo use Logstash to send data to Elasticsearch Serverless, you must be using:
- Logstash 8.10.1 or later
- Logstash Elasticsearch output plugin 11.18.0 or later
- Logstash Elasticsearch input plugin 4.18.0 or later
- Logstash Elasticsearch filter plugin 3.16.0 or later
Secure connection
editServerless Elasticsearch simplifies secure communication between Logstash and Elasticsearch.
Configure the Elasticsearch output plugin to use
cloud_id
and
api_key
.
No additional SSL configuration steps are needed.
API keys for connecting Logstash to Elasticsearch Serverless
editUse the Security: API key section in the UI to create an API key for securely connecting the Logstash Elasticsearch output to Elasticsearch Serverless. We recommend creating a unique API key per Logstash instance. You can create as many API keys as necessary.
When you set up your API keys, use the metadata option to tag each API key with details that are meaningful to you. This step makes managing multiple API keys easier.
After you generate an API key, add it to your Logstash Elasticsearch output plugin config file’s api_key
setting.
Here’s an example:
output { elasticsearch { api_key => "TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSowb0kQ0HWoA" } }
Migrating Elasticsearch data using Logstash
editYou can use Logstash to migrate data from self-managed Elasticsearch or Elasticsearch Service to Elasticsearch Serverless, or to migrate data from one Elasticsearch Serverless deployment to another.
Create a Logstash pipeline that includes the Elasticsearch input plugin and output plugin.
Configure the Elasticsearch input to point to your source deployment or instance, and configure the Elasticsearch output with the cloud_id
and api_key
settings for your target Elasticsearch Serverless instance.
If your origin index is using settings that aren’t supported in Serverless, then you might need to adjust your index settings.
Next steps
editCheck out the Logstash product page to see what Logstash can do for you. When you’re ready, dive into the Logstash documentation.