IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Glob Pattern Support Field References Deep Dive »
Elastic Docs ›Logstash Reference [8.13] ›Advanced Logstash Configurations

Converting Ingest Node Pipelines

edit

Converting Ingest Node Pipelines

edit

After implementing ingest pipelines to parse your data, you might decide that you want to take advantage of the richer transformation capabilities in Logstash. For example, you may need to use Logstash instead of ingest pipelines if you want to:

  • Ingest from more inputs. Logstash can natively ingest data from many other sources like TCP, UDP, syslog, and relational databases.
  • Use multiple outputs. Ingest node was designed to only support Elasticsearch as an output, but you may want to use more than one output. For example, you may want to archive your incoming data to S3 as well as indexing it in Elasticsearch.
  • Take advantage of the richer transformation capabilities in Logstash, such as external lookups.
  • Use the persistent queue feature to handle spikes when ingesting data (from Beats and other sources).

To make it easier for you to migrate your configurations, Logstash provides an ingest pipeline conversion tool. The conversion tool takes the ingest pipeline definition as input and, when possible, creates the equivalent Logstash configuration as output.

See Limitations for a full list of tool limitations.

Running the tool

edit

You’ll find the conversion tool in the bin directory of your Logstash installation. See Logstash Directory Layout to find the location of bin on your system.

To run the conversion tool, use the following command:

bin/ingest-convert.sh --input INPUT_FILE_URI --output OUTPUT_FILE_URI [--append-stdio]

Where:

  • INPUT_FILE_URI is a file URI that specifies the full path to the JSON file that defines the ingest node pipeline.
  • OUTPUT_FILE_URI is the file URI of the Logstash DSL file that will be generated by the tool.
  • --append-stdio is an optional flag that adds stdin and stdout sections to the config instead of adding the default Elasticsearch output.

This command expects a file URI, so make sure you use forward slashes and specify the full path to the file.

For example:

bin/ingest-convert.sh --input file:///tmp/ingest/apache.json --output file:///tmp/ingest/apache.conf

Limitations

edit
  • Painless script conversion is not supported.
  • Only a subset of available processors are supported for conversion. For processors that are not supported, the tool produces a warning and continues with a best-effort conversion.

Supported Processors

edit

The following ingest node processors are currently supported for conversion by the tool:

  • Append
  • Convert
  • Date
  • GeoIP
  • Grok
  • Gsub
  • Json
  • Lowercase
  • Rename
  • Set
« Glob Pattern Support Field References Deep Dive »