› › ›

Logstash

Logstash is an open source data collection engine with real-time pipelining capabilities. It supports a wide variety of data sources, and can dynamically unify data from disparate sources and normalize the data into destinations of your choice.

Logstash can collect data using a variety of Logstash input plugins, enrich and transform the data with Logstash filter plugins, and output the data to Elasticsearch using the Logstash Elasticsearch output plugin.

You can use Logstash to extend Beats for advanced use cases, such as data routed to multiple destinations or when you need to make your data persistent.

Logstash for Elasticsearch Serverless

Logstash is a powerful, versatile ETL (Extract, Transform, Load) engine that can play an important role in organizations of all sizes. Some capabilities and features for large, self-managed users aren’t appropriate for Serverless.

You’ll use the Logstash Elasticsearch output plugin to send data to Elasticsearch Serverless. Some differences to note between Elasticsearch Serverless and self-managed Elasticsearch:

Your logstash-output-elasticsearch configuration uses API keys to access Elasticsearch from Logstash. User-based security settings are ignored and may cause errors.
Elasticsearch Serverless uses data lifecycle management (DLM) instead of index lifecycle management (ILM). If you add ILM settings to your Elasticsearch output configuration, they are ignored and may cause errors.
Logstash monitoring for Serverless is available through the Logstash Integration in Elastic Observability.

Known issue

The logstash-output-elasticsearch hosts setting defaults to port :9200. Set the value to port :443 instead.

Requirements

edit

To use Logstash to send data to Elasticsearch Serverless, you must be using:

Logstash 8.10.1 or later
Logstash Elasticsearch output plugin 11.18.0 or later
Logstash Elasticsearch input plugin 4.18.0 or later
Logstash Elasticsearch filter plugin 3.16.0 or later

Secure connection

edit

Elasticsearch Serverless simplifies secure communication between Logstash and Elasticsearch. Configure the Elasticsearch output plugin to use cloud_id and api_key. No additional SSL configuration steps are needed.

API keys for connecting Logstash to Elasticsearch Serverless

edit

Use the Security: API key section in the UI to create an API key for securely connecting the Logstash Elasticsearch output to Elasticsearch Serverless. We recommend creating a unique API key per Logstash instance. You can create as many API keys as necessary.

When you set up your API keys, use the metadata option to tag each API key with details that are meaningful to you. This step makes managing multiple API keys easier.

After you generate an API key, add it to your Logstash Elasticsearch output plugin config file’s api_key setting. Here’s an example:

output {
  elasticsearch {
    api_key => "TiNAGG4BaaMdaH1tRfuU:KnR6yE41RrSowb0kQ0HWoA"
  }
}

Migrating Elasticsearch data using Logstash

edit

You can use Logstash to migrate data from self-managed Elasticsearch or Elasticsearch Service to Elasticsearch Serverless, or to migrate data from one Elasticsearch Serverless deployment to another.

Create a Logstash pipeline that includes the Elasticsearch input plugin and output plugin.

Configure the Elasticsearch input to point to your source deployment or instance, and configure the Elasticsearch output with the cloud_id and api_key settings for your target Elasticsearch Serverless instance.

If your origin index is using settings that aren’t supported in Serverless, then you might need to adjust your index settings.

Next steps

edit

Check out the Logstash product page to see what Logstash can do for you. When you’re ready, dive into the Logstash documentation.

« Upload a file Beats »