IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Enrich your data Enrich policy definition »
Elastic Docs ›Elasticsearch Guide [7.11] ›Ingest pipelines ›Enrich your data

Set up an enrich processor

edit

Set up an enrich processor

edit

To set up an enrich processor, follow these steps:

  1. Check the prerequisites.
  2. Add enrich data.
  3. Create an enrich policy.
  4. Execute the enrich policy.
  5. Add an enrich processor to an ingest pipeline.
  6. Ingest and enrich documents.

Once you have an enrich processor set up, you can update your enrich data and update your enrich policies.

The enrich processor performs several operations and may impact the speed of your ingest pipeline.

We strongly recommend testing and benchmarking your enrich processors before deploying them in production.

We do not recommend using the enrich processor to append real-time data. The enrich processor works best with reference data that doesn’t change frequently.

Prerequisites

edit

If you use Elasticsearch security features, you must have:

  • read index privileges for any indices used
  • The enrich_user built-in role

Add enrich data

edit

To begin, add documents to one or more source indices. These documents should contain the enrich data you eventually want to add to incoming documents.

You can manage source indices just like regular Elasticsearch indices using the document and index APIs.

You also can set up Beats, such as a Filebeat, to automatically send and index documents to your source indices. See Getting started with Beats.

Create an enrich policy

edit

After adding enrich data to your source indices, you can define an enrich policy. When defining the enrich policy, you should include at least the following:

  • A list of one or more source indices which store enrich data as documents
  • The policy type which determines how the processor matches the enrich data to incoming documents
  • A match field from the source indices used to match incoming documents
  • Enrich fields containing enrich data from the source indices you want to add to incoming documents

You can use this definition to create the enrich policy with the create enrich policy API.

Once created, you can’t update or change an enrich policy. See Update an enrich policy.

Execute the enrich policy

edit

Once the enrich policy is created, you can execute it using the execute enrich policy API to create an enrich index.

enrich policy index

The enrich index contains documents from the policy’s source indices. Enrich indices always begin with .enrich-*, are read-only, and are force merged.

Enrich indices should be used by the enrich processor only. Avoid using enrich indices for other purposes.

Add an enrich processor to an ingest pipeline

edit

Once you have source indices, an enrich policy, and the related enrich index in place, you can set up an ingest pipeline that includes an enrich processor for your policy.

enrich processor

Define an enrich processor and add it to an ingest pipeline using the create or update pipeline API.

When defining the enrich processor, you must include at least the following:

  • The enrich policy to use.
  • The field used to match incoming documents to the documents in your enrich index.
  • The target field to add to incoming documents. This target field contains the match and enrich fields specified in your enrich policy.

You also can use the max_matches option to set the number of enrich documents an incoming document can match. If set to the default of 1, data is added to an incoming document’s target field as a JSON object. Otherwise, the data is added as an array.

See Enrich for a full list of configuration options.

You also can add other processors to your ingest pipeline.

Ingest and enrich documents

edit

You can now use your ingest pipeline to enrich and index documents.

enrich process

Before implementing the pipeline in production, we recommend indexing a few test documents first and verifying enrich data was added correctly using the get API.

Update an enrich index

edit

Once created, you cannot update or index documents to an enrich index. Instead, update your source indices and execute the enrich policy again. This creates a new enrich index from your updated source indices and deletes the previous enrich index.

If wanted, you can reindex or update any already ingested documents using your ingest pipeline.

Update an enrich policy

edit

Once created, you can’t update or change an enrich policy. Instead, you can:

  1. Create and execute a new enrich policy.
  2. Replace the previous enrich policy with the new enrich policy in any in-use enrich processors.
  3. Use the delete enrich policy API to delete the previous enrich policy.
« Enrich your data Enrich policy definition »