IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Crawl a private network using a web crawler on Elastic Cloud Elasticsearch search for App Search Guide »
Elastic Docs › ›Guides

Curations

edit

Curations

edit

Curations allow search operators to customize search results for specific queries.

Use promoted documents to ensure that specified documents always match a query and receive the highest relevance scores. Imagine an ecommerce store with featured product results or a support forum with pinned results for the most popular queries. Similarly, use hidden documents to exclude particular documents from results.

Since 7.16.0, you can enable the new adaptive relevance feature to receive suggestions from App Search based on ongoing analysis of query and click data. These suggestions improve your curations and deliver more relevant results. Accept or reject each suggestion manually, or allow App Search to manage your curations automatically.

Readers familiar with curations and adaptive relevance can find solutions to specific problems within the sections under Using curations. Readers new to curations may want to begin with the Curations reference and Curations examples.

For API users, an API reference is available.

Using curations

edit

Manage curations using Kibana

edit

Manage curations using Kibana:

  1. Open: Enterprise Search → App Search → Engines → engine → Curations
  2. Create, list, view, update, and delete curations.

Watch the following video tutorial for details:


Manage curations using the Enterprise Search API

edit

Manage curations using the Enterprise Search API: Curations API reference.

Use the API to list, view, create, update, and delete curations.

Enable or disable adaptive relevance suggestions for an engine

edit

Enable or disable adaptive relevance suggestions for curations for each engine. This choice is stored as an adaptive relevance setting.

Use Kibana:

  1. Open: Enterprise Search → App Search → Engines → engine → Curations
  2. Choose the Settings tab, then select or de-select Enable suggestions.

Alternatively, use the Enterprise Search API to manage adaptive relevance settings. See Settings within the adaptive relevance API reference.

Manage adaptive relevance suggestions using Kibana

edit

Manage adaptive relevance suggestions for your curations using Kibana:

  1. Open: Enterprise Search → App Search → Engines → engine → Curations
  2. Choose the Overview tab if it isn’t already selected, and locate the Suggestions section.
  3. List and view adaptive relevance suggestions for your curations.
  4. Select a query from the list to view a specific suggestion.
  5. Accept, reject, automate, or disable suggestions for that query.

Manage adaptive relevance suggestions using the Enterprise Search API

edit

Manage adaptive relevance suggestions for your curations using the Enterprise Search API.

Use the adaptive relevance API operations: Adaptive relevance API reference (beta).

  • To limit results to curations suggestions, request suggestions for an engine where type is curation.
  • List suggestions or view specific suggestions.
  • For each suggestion: accept, reject, automate, or disable suggestions for that query.

Alternatively, use the curations API operations: Curations API reference.

  • List curations or view specific curations.
  • Each curation result includes suggestions information when present on the curation.

Accept all adaptive relevance suggestions for an engine

edit

App Search can automatically accept all adaptive relevance suggestions for your curations after creating each suggestion. In this mode, all new curations within the engine will be automated curations. This choice is stored as an adaptive relevance setting.

Use Kibana:

  1. Open: Enterprise Search → App Search → Engines → engine → Curations
  2. Choose the Settings tab, then select or de-select Automatically accept new suggestions.

Alternatively, use the Enterprise Search API to manage adaptive relevance settings. See Settings within the adaptive relevance API reference.

Manage adaptive relevance settings

edit

Manage adaptive relevance settings for each engine.

Manage some settings using Kibana:

Manage all settings using the Enterprise Search API. See Settings within the adaptive relevance API reference.

View adaptive relevance history

edit

View the following adaptive relevance history information:

  • Recent adaptive relevance changes
  • Recently rejected suggestions
  • Ignored queries

Use Kibana:

  1. Open: Enterprise Search → App Search → Engines → engine → Curations
  2. Choose the History tab, and locate any of the sections listed above.

Alternatively, view the adaptive relevance events logs. See View adaptive relevance events logs

View adaptive relevance events logs

edit

You can either use Kibana Logs or Kibana Discover:

Use Kibana Logs:

  1. Open: Observability → Logs
  2. In the search bar, select the search-relevance-suggestions dataset: event.dataset: search-relevance-suggestions
  3. Press enter, and you’ll see the filtered log stream for adaptive relevance suggestions related messages.

Use Kibana Discover:

Create an index pattern in Kibana for the adaptive relevance related indexes:

  1. Open: Stack Management → Kibana / Index Patterns → Create Index Pattern
  2. Click on "Advanced Settings" to check "Allow hidden and system indices".
  3. Set as "Name" the value .ent-search-search-relevance-suggestions-ecs-ilm-logs-*.
  4. Select @timestamp in the "Timestamp field" dropdown.
  5. Click "Create index pattern".

See create index patterns guide for more details.

Now you can select the newly created index pattern in Discover:

  1. Open Discover in the Kibana menu.
  2. Select the created index pattern (.ent-search-search-relevance-suggestions-ecs-ilm-logs-*) in the "Change index pattern" combo if it is not already displayed.

For a complete reference of all events, see Adaptive relevance events logs reference.

Convert an automated curation to a manual curation

edit

Convert an automated curation to a manual curation.

Use Kibana:

  1. Open: Enterprise Search → App Search → Engines → engine → Curations
  2. Select a curation to view its details, and choose Convert to manual curation.
Convert to manual curation

Convert a manual curation to an automated curation

edit

To convert a manual curation to an automated curation, you must wait for App Search to suggest a change to the curation.

Use Kibana:

  1. Open: Enterprise Search → App Search → Engines → engine → Curations
  2. Choose the Overview tab if it isn’t already selected, and locate the Suggestions section.
  3. Choose the suggestion that applies to the curation to visit its details screen.
  4. Choose ⋮ to expand the options menu, and choose Automate - always accept new suggestions for this query.
Automate curation

Curations reference

edit

For API users, an API reference is available.

Curation

edit

A curation is an object, associated with an engine, that represents customizations to the search results for one or more queries.

Each curation has the following attributes:

id

An opaque token created by App Search. Used for some API operations.

queries (required)

An array of strings, where each string is a search query whose results are modified by the curation.

Each query can belong to only one curation.

If adaptive relevance suggestions are enabled, use only one query per curation. App Search will suggest changes to only those curations that have a single query.

promoted

An array of strings, where each string is a document ID representing a promoted document.

To add a document to the promoted documents, it must exist within the engine.

When specifying documents for a curation on a meta engine, you must scope each document ID to its source engine. See Document IDs for meta engines.

Required when hidden is omitted.

hidden

An array of strings, where each string is a document ID representing a hidden document.

To add a document to the hidden documents, it must exist within the engine.

When specifying documents for a curation on a meta engine, you must scope each document ID to its source engine. See Document IDs for meta engines.

Required when promoted is omitted.

To manage curations, see:

App Search may create adaptive relevance suggestions for curations. In that context, it’s useful to distinguish between manual curations and automated curations. And it’s possible to convert between the two types. See:

Promoted documents

edit

Each curation may include a collection of promoted documents. These documents are always returned within the results for each query belonging to the curation, even if the documents do not match the query.

Promoted documents also receive higher _score values than any other results. Therefore, when results are sorted by _score (the default sort), promoted documents appear before all other results.

The promoted documents collection is ordered. When results are sorted by _score, promoted documents appear in the order in which they are stored on the curation, before all other results.

When results are sorted on a field other than _score, promoted documents are always included in the results, but their positions are determined by their value for the given sort field.

Similarly, when results are grouped on a field, promoted documents are always included in the results, but their positions are determined by their value for the given group field.

See Curations examples for examples that demonstrate promoted documents.

Hidden documents

edit

Each curation may include a collection of hidden documents. In contrast to promoted documents, hidden documents are never returned within the results for each query belonging to the curation, even if the documents match the query.

Since hidden documents are excluded from search results, the order of the collection is not important.

See Curations examples for examples that demonstrate hidden documents.

Adaptive relevance

edit

Adaptive relevance is a beta feature. Beta features are subject to change and are not covered by the support SLA of general release (GA) features. Elastic plans to promote this feature to GA in a future release.

Adaptive relevance is not available at all Elastic subscription levels. Refer to the Elastic subscriptions pages for Elastic Cloud and self-managed deployments.

Since 7.16.0, curations are powered by adaptive relevance. Enable this feature to receive suggestions from App Search based on ongoing analysis of query and click data.

App Search can create adaptive relevance suggestions that create, update, or delete your engines' curations. Accept these suggestions to improve your curations and deliver more relevant results.

You can enable, disable, and customize these suggestions using adaptive relevance settings.

Learn more about curations powered by adaptive relevance:

Use curations powered by adaptive relevance:

Adaptive relevance suggestions

edit

Enterprise Search 7.16.0 introduced curations powered by adaptive relevance. Enable this feature to receive suggestions from App Search to improve your curations and deliver more relevant results. Or, allow App Search to manage your curations automatically.

When enabled, App Search routinely analyzes queries within recent history with sufficient clicks. Based on that analysis, App Search creates suggestions to create, update, or delete curations that add or remove specific promoted documents.

Each suggestion applies to a specific query, so App Search will suggest changes only to those curations with a single query.

Each suggestion begins in a pending status.

Search operators can manage each suggestion using Kibana. Operators can:

  • Accept the suggestion. A curation will be created or updated with the suggestion.
  • Reject the suggestion. No curation will be created or updated. Updates to the suggestion will be highlighted again for approval.
  • Accept the suggestion and accept all future suggestions for the same curation. This converts the curation into an automated curation.
  • Disable the suggestion. The suggestion will be discarded and future suggestions for the same query will not be considered.

Alternatively, search operators can use the Enterprise Search API to directly manipulate the status of each suggestion. The possible statuses are:

  • pending: The suggestion is waiting to be accepted or discarded.
  • applied: The suggestion has been accepted and a curation has been created or updated with the suggestion.
  • automated: The suggestion has been accepted, and any future updates on the suggestion will be applied automatically.
  • rejected: The suggestion was rejected. Future updates to the suggestion will bring it back to pending state.
  • disabled: The suggestion has not been applied, and any future updates to the suggestion will be ignored.

Also refer to the following statuses diagram:

Adaptive relevance suggestions statuses

A pending suggestion may override a curation that has been created or modified manually if applied (or automated). When that would happen, a field override_manual_curation with value true will be returned as part of the suggestion information to warn about this.

Search operators can manage adaptive relevance settings to customize details of the algorithm or schedule for adaptive relevance suggestions. Search operators can also accept all suggestions for an engine to allow App Search to manage all new curations automatically.

To use adaptive relevance suggestions:

To manage adaptive relevance suggestions for an engine, see:

To view the adaptive relevance history for an engine, see:

Adaptive relevance settings

edit

Enterprise Search 7.16.0 introduced curations powered by adaptive relevance. Adaptive relevance settings are settings stored with each engine that allow search operators to customize the behavior of adaptive relevance suggestions for that engine. Use these settings to enable or disable adaptive relevance suggestions for an engine, accept all suggestions for an engine, or customize details of the algorithm or schedule for adaptive relevance suggestions for an engine.

Search operators can manage some settings using Kibana and all settings using the Enterprise Search API. See:

For a complete settings reference, see Settings within the adaptive relevance API reference.

Adaptive relevance events logs

edit

The adaptive relevance events logs record all events associated with adaptive relevance suggestions, including those that apply to curations. Use the logs to audit automated changes to your curations.

View the adaptive relevance events logs using Kibana Logs or Discover. See View adaptive relevance events logs.

Alternatively, view adaptive relevance history for curations using Kibana Curations. See View adaptive relevance history.

Automated curation

edit

An automated curation is a curation whose adaptive relevance suggestions have been pre-approved. App Search therefore applies new suggestions automatically as soon as they are obtained.

Search operators cannot change queries or promoted for an automated curation. These are managed exclusively by App Search. However, search operators can change hidden.

To change queries or promoted of an automated curation, first convert it to a manual curation. See Convert an automated curation to a manual curation.

Manual curation

edit

A manual curation is a curation whose adaptive relevance suggestions are not pre-approved. This is the default, standard curation type.

App Search may create adaptive relevance suggestions for a manual curation. Search operators may accept each suggestions manually, or they may pre-approve all future suggestions for a given curation. Pre-approving all suggestions converts the curation to an automated curation.

Curations examples

edit

The following sections use an ecommerce use case to demonstrate curations, promoted documents, and hidden documents.

An ecommerce analyst discovers that customers who search for winter coat usually buy the products Hooded parka and Down winter jacket. However, Hooded parka does not appear in the search results at all, and Down winter jacket is scored less relevant than a product that is purchased infrequently. Furthermore, the search results include a hat, which is not what searchers are looking for.

A search operator uses a curation to customize the results for the query winter coat. The curation ensures that Hooded parka is always returned and has the highest relevance score. The curation also scores Down winter jacket higher than the less popular product, Winter field coat. Finally, the curation excludes Knit winter hat from the results, since searchers consider it not relevant.

The following sections implement this scenario and also demonstrate the affect of sorting and grouping on curated results.

Within each section, expand the collapsed portions to view the relevant API requests and responses.

Create engine, schema, and documents

edit

Create an engine with name products to store documents representing products.

Create engine 
POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines

{
  "name": "products"
}

Define a schema for the documents within engine products. Documents in this engine have the following fields:

Field name Field type Description

id

id

A user-supplied ID for the product.

name

text

The name of the product.

price

number

The price of the product.

rating

number

The average customer rating of the product. One of: 1, 2, 3, 4, 5.
Create schema 
POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/products/schema

{
  "name": "text",
  "price": "number",
  "rating": "number"
}

Save the following documents within the products engine to use for the examples.

ID Name Price Rating

winter-field-coat

Winter field coat

75

4

down-winter-jacket

Down winter jacket

200

4

knit-winter-beanie

Knit winter beanie

10

5

hooded-parka

Hooded parka

150

5
Save documents 
POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/products/documents

[
  {
    "id": "winter-field-coat",
    "name": "Winter field coat",
    "price": 75,
    "rating": 4
  },
  {
    "id": "down-winter-jacket",
    "name": "Down winter jacket",
    "price": 200,
    "rating": 4
  },
  {
    "id": "knit-winter-beanie",
    "name": "Knit winter beanie",
    "price": 10,
    "rating": 5
  },
  {
    "id": "hooded-parka",
    "name": "Hooded parka",
    "price": 150,
    "rating": 5
  }
]

View search results

edit

Search for winter coat before curating the results. The search returns the following documents (with scores):

  1. Winter field coat (6.537158)
  2. Down winter jacket (0.49185246)
  3. Knit winter beanie (0.3164503)
Search for winter coat before curating results 
GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/products/search

{
  "query": "winter coat"
}
See full response (formatted) 
{
  "meta": {
    "alerts": [],
    "warnings": [],
    "precision": 2,
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 3,
      "size": 10
    },
    "engine": {
      "name": "products",
      "type": "default"
    },
    "request_id": "TPyQN9kIQOObgm8HU1Fjlg"
  },
  "results": [
    {
      "price": {
        "raw": 75
      },
      "name": {
        "raw": "Winter field coat"
      },
      "rating": {
        "raw": 4
      },
      "_meta": {
        "engine": "products",
        "score": 6.537158,
        "id": "winter-field-coat"
      },
      "id": {
        "raw": "winter-field-coat"
      }
    },
    {
      "price": {
        "raw": 200
      },
      "name": {
        "raw": "Down winter jacket"
      },
      "rating": {
        "raw": 4
      },
      "_meta": {
        "engine": "products",
        "score": 0.49185246,
        "id": "down-winter-jacket"
      },
      "id": {
        "raw": "down-winter-jacket"
      }
    },
    {
      "price": {
        "raw": 10
      },
      "name": {
        "raw": "Down winter jacket"
      },
      "rating": {
        "raw": 4
      },
      "_meta": {
        "engine": "products",
        "score": 0.49185246,
        "id": "down-winter-jacket"
      },
      "id": {
        "raw": "down-winter-jacket"
      }
    },
    {
      "price": {
        "raw": 10
      },
      "name": {
        "raw": "Knit winter beanie"
      },
      "rating": {
        "raw": 5
      },
      "_meta": {
        "engine": "products",
        "score": 0.3164503,
        "id": "knit-winter-beanie"
      },
      "id": {
        "raw": "knit-winter-beanie"
      }
    }
  ]
}

Curate results

edit

Recall the following suggestions from the ecommerce analyst to improve search results for winter coat:

  • Always return Hooded parka
  • Assign Hooded parka the highest relevance score
  • Assign Down winter jacket a higher relevance score than Winter field coat
  • Never return Knit winter hat

Create a curation with the following attributes:

Queries Promoted Hidden
  1. winter coat
  1. hooded-parka
  2. down-winter-jacket
  1. knit-winter-beanie
Create curation for query winter coat 
POST <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/products/curations

{
  "queries": [
    "winter coat"
  ],
  "promoted": [
    "hooded-parka",
    "down-winter-jacket"
  ],
  "hidden": [
    "knit-winter-beanie"
  ]
}

Now, search again to see how the curation affects the results:

  1. Hooded parka (1.7014124e+38)
  2. Down winter jacket (1.7014122e+38)
  3. Winter field coat (6.537158)

The following has changed:

  • Hooded parka is included
  • Knit winter hat is excluded
  • Hooded parka and Down winter jacket have higher scores than Winter field coat
Search for winter coat after curating results 
GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/products/search

{
  "query": "winter coat"
}
See full response (formatted) 
{
  "meta": {
    "alerts": [],
    "warnings": [],
    "precision": 2,
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 3,
      "size": 10
    },
    "engine": {
      "name": "products",
      "type": "default"
    },
    "request_id": "qQq_gg9PT_migDYWsTIPyw"
  },
  "results": [
    {
      "price": {
        "raw": 150
      },
      "name": {
        "raw": "Hooded parka"
      },
      "rating": {
        "raw": 5
      },
      "_meta": {
        "engine": "products",
        "score": 1.7014124e+38,
        "id": "hooded-parka"
      },
      "id": {
        "raw": "hooded-parka"
      }
    },
    {
      "price": {
        "raw": 200
      },
      "name": {
        "raw": "Down winter jacket"
      },
      "rating": {
        "raw": 4
      },
      "_meta": {
        "engine": "products",
        "score": 1.7014122e+38,
        "id": "down-winter-jacket"
      },
      "id": {
        "raw": "down-winter-jacket"
      }
    },
    {
      "price": {
        "raw": 75
      },
      "name": {
       "raw": "Down winter jacket"
      },
      "rating": {
        "raw": 4
      },
      "_meta": {
        "engine": "products",
        "score": 1.7014122e+38,
        "id": "down-winter-jacket"
      },
      "id": {
        "raw": "down-winter-jacket"
      }
    },
    {
      "price": {
        "raw": 75
      },
      "name": {
        "raw": "Winter field coat"
      },
      "rating": {
        "raw": 4
      },
      "_meta": {
        "engine": "products",
        "score": 6.537158,
        "id": "winter-field-coat"
      },
      "id": {
        "raw": "winter-field-coat"
      }
    }
  ]
}

Sort curated results

edit

The following example demonstrates the affect of sorting on curated results.

Sort the results by price, ascending (low to high):

  1. Winter field coat (75)
  2. Hooded parka (150)
  3. Down winter jacket (200)

Observe the following:

  • Hooded parka is still included, despite not matching the query
  • Hooded parka and Down winter jacket now appear after Winter field coat, because the documents are sorted by price rather than _score
Sort curated results 
GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/products/search

{
  "query": "winter coat",
  "sort": { "price": "asc" }
}
See full response (formatted) 
{
  "meta": {
    "alerts": [],
    "warnings": [],
    "precision": 2,
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 3,
      "size": 10
    },
    "engine": {
      "name": "products",
      "type": "default"
    },
    "request_id": "PjSYjwuETQ2YyC05VmynHw"
  },
  "results": [
    {
      "price": {
        "raw": 75
      },
      "name": {
        "raw": "Winter field coat"
      },
      "rating": {
        "raw": 4
      },
      "_meta": {
        "engine": "products",
        "score": null,
        "id": "winter-field-coat"
      },
      "id": {
        "raw": "winter-field-coat"
      }
    },
    {
      "price": {
        "raw": 150
      },
      "name": {
        "raw": "Hooded parka"
      },
      "rating": {
        "raw": 5
      },
      "_meta": {
        "engine": "products",
        "score": null,
        "id": "hooded-parka"
      },
      "id": {
        "raw": "hooded-parka"
      }
    },
    {
      "price": {
        "raw": 200
      },
      "name": {
        "raw": "Down winter jacket"
      },
      "rating": {
        "raw": 4
      },
      "_meta": {
        "engine": "products",
        "score": null,
        "id": "down-winter-jacket"
      },
      "id": {
        "raw": "down-winter-jacket"
      }
    }
  ]
}

Group curated results

edit

The following example demonstrates the affect of grouping on curated results.

Group the results by rating:

5:

  1. Hooded parka

4:

  1. Winter field coat
  2. Down winter jacket

Observe the following:

  • Hooded parka is still included, despite not matching the query
  • Down winter jacket now appears after Winter field coat, because the documents are organized in groups by their rating rather than sorted by score
Group curated results 
GET <ENTERPRISE_SEARCH_BASE_URL>/api/as/v1/engines/products/search

{
  "query": "winter coat",
  "group": { "field": "rating" }
}
See full response (formatted) 
{
  "meta": {
    "alerts": [],
    "warnings": [],
    "precision": 2,
    "page": {
      "current": 1,
      "total_pages": 1,
      "total_results": 2,
      "size": 10
    },
    "engine": {
      "name": "products",
      "type": "default"
    },
    "request_id": "XegjDUewRWOj-5sr70DBYw"
  },
  "results": [
    {
      "name": {
        "raw": "Hooded parka"
      },
      "price": {
        "raw": 150
      },
      "rating": {
        "raw": 5
      },
      "_meta": {
        "engine": "products",
        "score": 1.7014124e+38,
        "id": "hooded-parka"
      },
      "id": {
        "raw": "hooded-parka"
      },
      "_group": [],
      "_group_key": 5
    },
    {
      "name": {
        "raw": "Down winter jacket"
      },
      "price": {
        "raw": 200
      },
      "rating": {
        "raw": 4
      },
      "_meta": {
        "engine": "products",
        "score": 1.7014122e+38,
        "id": "down-winter-jacket"
      },
      "id": {
        "raw": "down-winter-jacket"
      },
      "_group": [
        {
          "name": {
            "raw": "Winter field coat"
          },
          "price": {
            "raw": 75
          },
          "rating": {
            "raw": 4
          },
          "_meta": {
            "engine": "products",
            "score": 6.537158,
            "id": "winter-field-coat"
          },
          "id": {
            "raw": "winter-field-coat"
          }
        }
      ],
      "_group_key": 4
    }
  ]
}
« Crawl a private network using a web crawler on Elastic Cloud Elasticsearch search for App Search Guide »