IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Log settings guide Precision tuning (beta) »
Elastic Docs › ›Guides

Meta Engines Guide

edit

Meta Engines Guide

edit

What is a Meta Engine

edit

This feature is not available for all Elastic subscription levels. Refer to the subscriptions pages for Elastic Cloud and Elastic Stack. To change your subscription level or start a trial, see Subscriptions and features.

A Meta Engine is an Engine that has no Documents of its own. Instead, it combines multiple other Engines so that they can be searched together as if they were a single Engine.

The Engines that comprise a Meta Engine are referred to as Source Engines.

Example: Two Engines, "western-national-parks" and "eastern-national-parks" can be combined in a Meta Engine named "national-parks" so that they can be searched as one data-set.

GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/parks/search

{
  "results": [
    {
      "id": "eastern-national-parks|park_shenandoah",
      "_meta": {
        "id": "park_shenandoah",
        "engine": "eastern-national-parks"
      }
    },
    {
      "id": "western-national-parks|park_yosemite",
      "_meta": {
        "id": "park_yosemite",
        "engine": "western-national-parks"
      }
    }
  ]
}

Why use a Meta Engine?

edit

Meta Engines allow users to create a document-less Engine that searches over a set of existing Engines, with search settings that are separate from those Source Engines. They enable comprehensive search over a collection of documents that have subsets with different search setting requirements.

For example, search experiences in large organizations often require documents from multiple business units, with standalone sites with Relevance Tunings specific to their needs. They still need to be part of a company-wide search experience, and organization-wide search priorities that require Curations or Relevance Tunings that differ from specific business-unit needs. Meta Engines allow both to exist together with a single engine-source-of-truth.

Permissions also come into play - one unit might not want other units to be able to write new documents into their Engine, for example. Remember, anyone with access to a Meta Engine has read access to all the Source Engines attached to it.

Pitfalls

edit

Meta Engines are useful but there are many other cases where a single Engine is a better choice. Here are some examples where Meta Engines are not the right solution:

  • Large amounts of fields: Meta Engines aren’t a way to expand the number of fields for a single set of documents. The best search results appear after condensing fields to just the essentials within one Engine. Remember that more fields results in slower queries.
  • A way to avoid identifying search language: If you have the same content in multiple languages, indexing all of them using the Universal language choice works well. If you do have different Engines for each language, a much more effective pattern is to identify the language — e.g. via browser language settings or language detection — and direct the query to the correct Engine.
  • Adjustments to Relevance Tuning for a single Engine: Search settings are controllable from the query side; consider passing Search Fields query parameters instead.

Things to Remember

edit

There are a few important things to remember that will help make Meta Engines work for you:

  • Unified Schemas: Meta Engine schemas are the union of all Source Engines schemas. Therefore, if two Source Engines share a field, their names must match - if the names are different, they will be treated as a different field! As well, the fields need to have the same type. See more below on how schema conflicts are treated.
  • Schema changes bubble up: Changes to a Source Engine’s schema will bubble up to a Meta Engine - so when changing field names or types in Source Engines, remember those are passed to the Meta Engine. Make sure to adapt any queries that rely on specific fields or their types.
  • Settings changes do not bubble up: Things like Curations, Relevance Tuning, or Result Settings that are made in a Source Engine will not affect queries on the Meta Engine.
  • Permissions: Meta Engine users have read permissions to all the docs visible to the Meta Engine. Conversely, having write permissions to a Meta Engine doesn’t grant write permission to Source Engines.
  • Documents and Schemas within Meta Engines are Read-Only - Meta Engines contain no Documents of their own. Because of this, no Document management or Schema updates can be performed directly on a Meta Engine, and instead must be performed on individual Source Engines.

Creating Meta Engines in the UI

edit

You must have the "Admin" or "Owner" role to create a Meta Engine.

Navigate to the Meta Engine page using the Meta Engines link in the main navigation. This is the main page where Meta Engines are managed.

Click on Create a Meta Engine. Name your Meta Engine and pick 2 or more existing Engines to use as Source Engines.

After completing this form your Meta Engine will be available for use immediately.

Changing Source Engines in Meta Engines

edit

Source Engines can be freely changed after a Meta Engine has been created.

Within the navigation of a Meta Engine, navigate to the Engines page via the Engines link. This page lists the current Source Engines for your Meta Engine and allows you to add and remove them.

Use caution when adding a Source Engine to an existing Meta Engine. If adding a new Source Engine would create a Schema Conflict, that field will become disabled and disappear from search query results. See more below on how schema conflicts are treated.

Managing Meta Engines with the API

edit

Meta Engines can also be created and managed through the API.

See the Meta Engines API reference for more information.

Meta Engine Schema and Conflicts

edit

Schema field type conflicts

edit

A meta engine’s schema is the combination of the source engines' schemas. This can be seen on the meta Engine’s schema page, which shows the schema fields along with the source engines those fields originate from.

If two or more source engines define a field with the same name and the same field type, then they will be treated as a single field within the schema.

Example: If two source engines, western-national-parks and eastern-national-parks, each have a "name" field of type "text", the meta engines schema will also have a "name" field of type "text".

If however, two or more source engines have a field with the same name and a different field type, then that is considered a schema field type conflict and that field will be completely disabled; meaning it will not be returned in search results, it cannot be filtered on, it cannot be used for boosting, etc.

Example: If two source engines, western-national-parks and eastern-national-parks, each have a "visitors" field, but western-national-parks defines it as "text" and eastern-national-parks defines it as "number", then the "visitors" field will be disabled.

Because of the "disabling" nature of schema conflicts, it is recommended to use caution when editing source engines.

Resolving a schema field type conflict
edit

A schema field type conflict can be resolved by navigating to the source engine directly in the UI and updating the field type of the conflicting field to match that of the other source engines.

Example: To resolve a schema field type conflict on the "visitors" field between the western-national-parks and eastern-national-parks source engines, you would navigate to the western-national-parks engine’s schema page and update the "visitors" field to type "number", or vice versa.

Mapping conflicts

edit

When using an Elasticsearch index engine in a meta engine, you may see the following error:

{
   "errors" : [
      "Source engines have incompatible field mappings for the following fields: foo"
   ]
}

This means the Elasticsearch mappings of the multiple foo fields are not compatible with each other.

Text field mapping conflicts examples
edit

engine-1: An App Search engine with the following schema:

{ "foo": "text" }

engine-2: An Elasticsearch index engine with the following Elasticsearch mapping:

{
  "foo": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    }
  }
}

engine-3: An Elasticsearch index engine with the following Elasticsearch mapping:

{
  "properties": {
    "foo": {
      "type": "keyword",
      "fields": {
        "stem": {
          "type": "text"
        }
      }
    }
  }
}

For each index App Search infers a keyword subfield automatically to be used in search request to build features like facets, filters, …​ In this situation, there isn’t a consistent name for the subfield across the indices:

  • In engine-1 the subfield name is foo.enum
  • In engine-2 the subfield name is foo.keyword
  • In engine-3 the subfield name is foo

To resolve the mapping conflict, reindex data for engine-2 and engine-3 into new indices that follow the Enterprise Search mapping conventions.

Non-text field mapping conflicts examples
edit

engine-1: An App Search engine with the following schema:

{ "foo": "number" }

engine-2: An Elasticsearch index engine with the following Elasticsearch mapping:

{
  "properties": {
    "foo": {
      "type": "float"
    }
  }
}

For each index App Search infers a numeric subfield to be used in search request to build features like facets, filters, …​ In this situation, there isn’t a consistent name for the subfield across the indices:

  • In engine-1 the subfield name is foo.float
  • In engine-2 the subfield name is foo

There is currently no workaround to solve such a mapping conflict.

Scoped Document IDs

edit

Documents within Meta Engines always use a Scoped ID, rather than a typical Document ID. A Scoped ID is the combination of Engine Name and Document ID with a pipe delimiter.

{Engine Name}|{Document ID}

Example: eastern-national-parks|park_shenandoah

Keep this in mind, because you will need to use a Scoped ID when using API endpoints like the Curations endpoint for Meta Engines, which requires a Document ID as a parameter.

« Log settings guide Precision tuning (beta) »