Customize crawler field values using an ingest pipeline

In this guide, you’ll learn how to use ingest pipelines to remove unwanted content from webpages you’ve crawled using the Elastic web crawler. Most websites contain content that is not needed for a search experience, such as navigation links, headers and footers, and other boilerplate.

If you can edit the HTML for the web pages you’re crawling, you can use Meta tags and data attributes to extract custom fields. However, if you don’t control the source HTML, you’ll need a workaround to remove this content from your documents.

In this guide, we’ll show you another option, using ingest pipelines. Ingest pipelines are a native Elasticsearch feature for transforming data. We’ll use a custom ingest pipeline to modify the values of specific fields in our documents, before they are written to Elasticsearch. The advantage of this approach is that we can manage everything from the Kibana UI.

In this example, imagine we are creating a search experience for a news website. For this example, we chose the https://www.theskimm.com. We use the Elastic web crawler to crawl the website and index the content. However, the website contains a lot of boilerplate content that we don’t want to include in our search results. We know this because we’ve crawled the website and inspected the fields in the Elasticsearch documents created by the crawler.

To remove this content, we’ll use an ingest pipeline to modify the values of specific fields in our documents.

In this example we will:

To use Enterprise Search features you need a subscription, a deployment, and a user. You get all three with an Elastic Cloud deployment. If you’re brand new to Elastic start a free Elastic Cloud trial.

Within Advanced settings, ensure the Enterprise Search Size per zone is set to at least 4 GB RAM.

First we need to create an Elasticsearch index and configure the web crawler to crawl the domain:

The crawl shouldn’t take long (less than one minute), because we’ll only be indexing around 250 documents. Once the crawl is complete, take a minute to inspect a few documents. Go to the Documents tab and click on a few documents to inspect the content.

You’ll notice a few things:

We don’t want this redundant content in our search results, so we’ll need to remove it. To do this, we’ll create a custom ingest pipeline that uses a set of processors to find and remove patterns from specific fields.

When you create an index in Enterprise Search, a default ingest pipeline is set up with several processors, to optimize your content for search. This pipeline is called ent-search-generic-ingestion. Enterprise Search also enables you to easily create index-specific ingest pipelines for custom processing. We’ll create a new pipeline with three processors targeting specific fields in our documents.

To create a custom pipeline:

An ingest pipeline is a list of processors that are applied to a document in order. The pipeline we created is empty, so now we’ll need to add some processors.

The <index-name>@custom pipeline is empty by default. We’ll add processors in the Kibana UI.

For this exercise we’ll use the Gsub processor. The Gsub processor allows you to replace (or simply delete) substrings within a field. We’ll use it to remove the header/footer and title boilerplate from our document fields.

We need to configure three Gsub processors:

The following table shows the configuration for each processor. When you configure your three processors, you’ll use the Field and Pattern values from this table.

Because we’re adding three Gsub processors, you’ll need to repeat these steps three times.

To add a processor:

Next, we’ll configure each processor to find and remove matched patterns from specific fields. Again, you’ll need to repeat this operation three times, once for each processor.

Follow these steps, copying the values in the table above to configure each processor.

Our pipeline has been configured, but we should test it to make sure the three processors work as expected. This allows us to correct any problems before we start crawling. This is important, particularly if you’re crawling a large number of webpages.

You can test your custom pipeline in the Kibana UI:

Now that we’ve tested our pipeline, we can start crawling our webpages.

If you’ve followed along, well done! You’ve successfully configured your Enterprise Search deployment to customize how webpages are indexed. You’ve learned the basics of a powerful set of tools for creating custom search experiences. Use this guide as a blueprint for more complex use cases. There are currently around 40 available processors that are ready to use. See the Elasticsearch documentation for the full list.