Ingest pipelines

This content applies to:

Ingest pipelines let you perform common transformations on your data before indexing. For example, you can use pipelines to remove fields, extract values from text, and enrich your data.

A pipeline consists of a series of configurable tasks called processors. Each processor runs sequentially, making specific changes to incoming documents. After the processors have run, Elasticsearch adds the transformed documents to your data stream or index.

In Project settings → Management → Ingest Pipelines, you can:

To create a pipeline, click Create pipeline → New pipeline. For an example tutorial, see Example: Parse logs.

The New pipeline from CSV option lets you use a file with comma-separated values (CSV) to create an ingest pipeline that maps custom data to the Elastic Common Schema (ECS). Mapping your custom data to ECS makes the data easier to search and lets you reuse visualizations from other data sets. To get started, check Map custom data to ECS.

Before you use a pipeline in production, you should test it using sample documents. When creating or editing a pipeline in Ingest Pipelines, click Add documents. In the Documents tab, provide sample documents and click Run the pipeline: