This documentation contains work-in-progress information for future Elastic Stack and Cloud releases. Use the version selector to view supported release docs. It also contains some Elastic Cloud serverless information. Check out our serverless docs for more details.

« Elasticsearch search for App Search Guide Create Elasticsearch index engines »

› ›

Elasticsearch index engines

edit

Elasticsearch index engines

edit

Since 8.2.0, Elastic Enterprise Search allows you to generate Engines based on existing Elasticsearch indices. With these engines you can use Enterprise Search features on data that has already been ingested into Elasticsearch.

Elasticsearch index engines features

edit

The following table compares Enterprise Search managed engines with Elasticsearch index engines:

Feature	Engine	Elasticsearch Index Engine
Documents API	Get, List, Create, Update, Delete	Get, List
Schema API	List, Update	List
Search API precision (beta), Precision tuning (beta)	Supported	Supported for specific fields, see Elasticsearch engines text field conventions
Usable as a source engine in meta engines	Supported	Supported
Synonyms	Supported	Supported
Curations	Supported	Supported
API logs	Supported	Supported
Analytics	Supported	Supported
Adaptive relevance	Supported	Supported
Query suggestions	Supported	Supported
Web Crawler	App Search Web Crawler	Elastic Web Crawler

Elasticsearch index engines supported field types

edit

App Search engines do not support all Elasticsearch field types. Enterprise Search will ignore unsupported Elasticsearch field types.

Enterprise Search will not modify your data and index mappings.

Unsupported field types will not be removed or altered and will remain available for Elasticsearch features and APIs.

Enterprise Search supports the following Elasticsearch field types:

Elasticsearch field type	Corresponding Enterprise Search field type
keyword	text
constant_keyword	text
wildcard	text
text	text
geo_point	geolocation
date	date
date_nanos	date
integer	number
long	number
short	number
byte	number
double	number
float	number
half_float	number
scaled_float	number
unsigned_long	number

See Supported Features, By Field Type for a full App Search field types reference.

Unlike the text field type in Enterprise Search, not all Elasticsearch text fields above have the same features as Enterprise Search text fields: keyword, constant_keyword, wildcard can be filtered, sorted, grouped and faceted. text fields cannot. Having a subfield for text fields will provide those capabilities to the field.

Object fields

edit

Elasticsearch has an object field type intended to support JSON inner objects.

In the following mapping the fields manager and manager.name are object fields:

{
  "mappings": {
    "properties": {
      "manager": {
        "properties": {
          "age":  { "type": "integer" },
          "name": {
            "properties": {
              "first": { "type": "text" },
              "last":  { "type": "text" }
            }
          }
        }
      }
    }
  }
}

When using such a mapping, subfields of the object fields are added to the schema of the Elasticsearch index engine:

{
  "manager.age": "number",
  "manager.name.first": "text",
  "manager.name.last": "text"
}

Object fields features

edit

Use subfields of an object just like any other standard field in Search API features, including in filters, facets, sorting, and groups.

Object fields in search results

edit

Object fields are rendered like any other field in search results:

{
  "manager.age": {
    "raw": 30
  },
  "manager.name.first": {
    "raw": "John"
  },
  "manager.name.last": {
    "raw": "Smith"
  }
}

Elasticsearch flattens object field data.

Internally, Elasticsearch flattens object field data into multi-value fields.

Given the following document:

{
  "manager": [
    {
      "name": {
        "first" : "John",
        "last" :  "Smith"
      }
    },
    {
      "name": {
        "first" : "Alice",
        "last" :  "White"
      }
    }
  ]
}

Elasticsearch will store it internally this way:

{
  "manager.name.first": ["John", "Alice"],
  "manager.mame.last": ["Smith", "White"]
}

In order to stay consistent with how objects are internally stored, the App Search Search API returns flattened object fields:

"manager.name.first": {
    "raw": ["John", "Alice"]
  },
  "manager.name.last": {
    "raw": ["Smith", "White"]
  }

To keep the independence of each object in the array and not lose the association between Alice and White, use the nested field type instead of the object field type.

Nested fields

edit

Elasticsearch has a nested field type intended to support JSON inner objects. Unlike the object field type, Elasticsearch does not flatten nested object values into multi-value fields.

In the following mapping the field manager is a nested object field:

{
  "mappings": {
    "properties": {
      "manager": {
        "type": "nested",
        "properties": {
          "age":  { "type": "integer" },
          "name": {
            "properties": {
              "first": { "type": "text" },
              "last":  { "type": "text" }
            }
          }
        }
      }
    }
  }
}

When using such a mapping, the nested field and its subfields are added to the Elasticsearch index engine schema in a flattened form:

{
  "manager": "nested",
  "manager.age": "number",
  "manager.name.first": "text",
  "manager.name.last": "text"
}

Nested fields filters

edit

It is possible to use subfields of a nested field as filters in the Search API.

{
  "query": "",
  "filters": {
    "manager.name.first": "Rich"
  }
}

Nested fields limitations

edit

Nested fields cannot be used as search_fields in the Search API. Nested fields do not contribute to full-text search.

Additionally, nested fields and their subfield are not usable with the following features of the Search API:

Searching, faceting, boosting, and sorting with nested fields

To include a nested field’s textual value in a full-text search query, you can copy the value of the subfield into the root of the document by remapping your Elasticsearch index using copy_to.

{
  "mappings": {
    "properties": {
      "manager_first_name": {
        "type": "text"
      },
      "manager": {
        "type": "nested",
        "properties": {
          "age":  { "type": "integer" },
          "name": {
            "properties": {
              "first": {
                "type": "text",
                "copy_to": "manager_first_name"
              },
              "last": {
                "type": "text"
              }
            }
          }
        }
      }
    }
  }
}

By doing so, you create a new manager_first_name field containing all the values of manager.name.first. Because the copied field exists outside of the nested object it can be used as a full-text search field.

You can use this technique to build facets, boosts, sort orders or groups from nested object subfields.

Nested fields in search results

edit

Unlike with the object field type, App Search does not flatten nested object values into multi-value fields in search results. App Search flattens objects but preserves the overall structure of the original document.

Given the following document:

{
  "manager": [
    {
      "name": {
        "first" : "John",
        "last" :  "Smith"
      }
    },
    {
      "name": {
        "first" : "Alice",
        "last" :  "White"
      }
    }
  ]
}

App Search will render it this way in Search API results:

{
  "manager": [
    {
      "name.first": { "raw": "John" },
      "name.last": { "raw": "Smith" }
    },
    {
      "name.first": { "raw": "Alice" },
      "name.last": { "raw": "White" }
    }
  ]
}

Because nested object subfields cannot be used in full-text search, using them as a snippet in search results is not possible. App Search will raise an error if you use a nested field or subfield as a snippet.

« Elasticsearch search for App Search Guide Create Elasticsearch index engines »