Elasticsearch index engines

edit

Elasticsearch index engines

edit

Since 8.2.0, Elastic Enterprise Search allows you to generate Engines based on existing Elasticsearch indices. With these engines you can use Enterprise Search features on data that has already been ingested into Elasticsearch.

Elasticsearch index engines features

edit

The following table compares Enterprise Search managed engines with Elasticsearch index engines:

Feature

Engine

Elasticsearch Index Engine

Documents API

Get, List, Create, Update, Delete

Get, List

Schema API

List, Update

List

Search API precision (beta), Precision tuning (beta)

Supported

Supported for specific fields, see Elasticsearch engines text field conventions

Usable as a source engine in meta engines

Supported

Supported

Synonyms

Supported

Supported

Curations

Supported

Supported

API logs

Supported

Supported

Analytics

Supported

Supported

Adaptive relevance

Supported

Supported

Query suggestions

Supported

Supported

Web Crawler

App Search Web Crawler

Elastic Web Crawler

Elasticsearch index engines supported field types

edit

App Search engines do not support all Elasticsearch field types. Enterprise Search will ignore unsupported Elasticsearch field types.

Enterprise Search will not modify your data and index mappings.

Unsupported field types will not be removed or altered and will remain available for Elasticsearch features and APIs.

Enterprise Search supports the following Elasticsearch field types:

Elasticsearch field type

Corresponding Enterprise Search field type

keyword

text

constant_keyword

text

wildcard

text

text

text

geo_point

geolocation

date

date

date_nanos

date

integer

number

long

number

short

number

byte

number

double

number

float

number

half_float

number

scaled_float

number

unsigned_long

number

See Supported Features, By Field Type for a full App Search field types reference.

Unlike the text field type in Enterprise Search, not all Elasticsearch text fields above have the same features as Enterprise Search text fields: keyword, constant_keyword, wildcard can be filtered, sorted, grouped and faceted. text fields cannot. Having a subfield for text fields will provide those capabilities to the field.

Object fields

edit

Elasticsearch has an object field type intended to support JSON inner objects.

In the following mapping the fields manager and manager.name are object fields:

{
  "mappings": {
    "properties": {
      "manager": {
        "properties": {
          "age":  { "type": "integer" },
          "name": {
            "properties": {
              "first": { "type": "text" },
              "last":  { "type": "text" }
            }
          }
        }
      }
    }
  }
}

When using such a mapping, subfields of the object fields are added to the schema of the Elasticsearch index engine:

{
  "manager.age": "number",
  "manager.name.first": "text",
  "manager.name.last": "text"
}
Object fields features
edit

Use subfields of an object just like any other standard field in Search API features, including in filters, facets, sorting, and groups.

Object fields in search results
edit

Object fields are rendered like any other field in search results:

{
  "manager.age": {
    "raw": 30
  },
  "manager.name.first": {
    "raw": "John"
  },
  "manager.name.last": {
    "raw": "Smith"
  }
}

Elasticsearch flattens object field data.

Internally, Elasticsearch flattens object field data into multi-value fields.

Given the following document:

{
  "manager": [
    {
      "name": {
        "first" : "John",
        "last" :  "Smith"
      }
    },
    {
      "name": {
        "first" : "Alice",
        "last" :  "White"
      }
    }
  ]
}

Elasticsearch will store it internally this way:

{
  "manager.name.first": ["John", "Alice"],
  "manager.mame.last": ["Smith", "White"]
}

In order to stay consistent with how objects are internally stored, the App Search Search API returns flattened object fields:

"manager.name.first": {
    "raw": ["John", "Alice"]
  },
  "manager.name.last": {
    "raw": ["Smith", "White"]
  }

To keep the independence of each object in the array and not lose the association between Alice and White, use the nested field type instead of the object field type.

Nested fields

edit

Elasticsearch has a nested field type intended to support JSON inner objects. Unlike the object field type, Elasticsearch does not flatten nested object values into multi-value fields.

In the following mapping the field manager is a nested object field:

{
  "mappings": {
    "properties": {
      "manager": {
        "type": "nested",
        "properties": {
          "age":  { "type": "integer" },
          "name": {
            "properties": {
              "first": { "type": "text" },
              "last":  { "type": "text" }
            }
          }
        }
      }
    }
  }
}

When using such a mapping, the nested field and its subfields are added to the Elasticsearch index engine schema in a flattened form:

{
  "manager": "nested",
  "manager.age": "number",
  "manager.name.first": "text",
  "manager.name.last": "text"
}
Nested fields filters
edit

It is possible to use subfields of a nested field as filters in the Search API.

{
  "query": "",
  "filters": {
    "manager.name.first": "Rich"
  }
}
Nested fields limitations
edit

Nested fields cannot be used as search_fields in the Search API. Nested fields do not contribute to full-text search.

Additionally, nested fields and their subfield are not usable with the following features of the Search API:

Searching, faceting, boosting, and sorting with nested fields

To include a nested field’s textual value in a full-text search query, you can copy the value of the subfield into the root of the document by remapping your Elasticsearch index using copy_to.

{
  "mappings": {
    "properties": {
      "manager_first_name": {
        "type": "text"
      },
      "manager": {
        "type": "nested",
        "properties": {
          "age":  { "type": "integer" },
          "name": {
            "properties": {
              "first": {
                "type": "text",
                "copy_to": "manager_first_name"
              },
              "last": {
                "type": "text"
              }
            }
          }
        }
      }
    }
  }
}

By doing so, you create a new manager_first_name field containing all the values of manager.name.first. Because the copied field exists outside of the nested object it can be used as a full-text search field.

You can use this technique to build facets, boosts, sort orders or groups from nested object subfields.

Nested fields in search results
edit

Unlike with the object field type, App Search does not flatten nested object values into multi-value fields in search results. App Search flattens objects but preserves the overall structure of the original document.

Given the following document:

{
  "manager": [
    {
      "name": {
        "first" : "John",
        "last" :  "Smith"
      }
    },
    {
      "name": {
        "first" : "Alice",
        "last" :  "White"
      }
    }
  ]
}

App Search will render it this way in Search API results:

{
  "manager": [
    {
      "name.first": { "raw": "John" },
      "name.last": { "raw": "Smith" }
    },
    {
      "name.first": { "raw": "Alice" },
      "name.last": { "raw": "White" }
    }
  ]
}

Because nested object subfields cannot be used in full-text search, using them as a snippet in search results is not possible. App Search will raise an error if you use a nested field or subfield as a snippet.