Blog

Elasticsearch query rules are now generally available

Introducing the general availability of query rules

Search Relevance
KD
By: Kathleen DeRusso

On

Query rules allow fine-grained, contextually specific solutions to alter search results for specific queries or search use cases. This can be helpful for campaigns that require branded or sponsored results to be pinned to the top of the search result lists for specific keywords, but can also be helpful for head queries where you simply need to "fix" the top result.

Introduction

We’re happy to announce that query rules are generally available in our serverless offering, and will be generally available starting with stack version 8.15.0. Query rules were first introduced in 8.10.0 as a technical preview feature, and allow index maintainers to curate specific documents to pin at the top of results based on contextual query-entered criteria.

How do query rules work?

Query rules are rules that you define based on specific query metadata. You start by defining a query ruleset that identifies documents to promote for certain metadata sent in with the query. Then at search time, you send that metadata in along with a rule query. If the metadata sent in the rule query matches any rules, those rules will be applied to your result set.

New query rule features

We added some features to query rules as we marched along the path to general availability.

A quick summary of these changes:

  • We’ve renamed the rule query from rule_query to rule, to be more consistent with our other API calls
  • We now support specifying multiple rulesets in a single rule query
  • We’ve expanded our query ruleset management APIs to support managing individual query rules

What does this look like?

Let’s say we have an index with dog breed information containing two fields: dog_breed and advert. We want to set up rules that pin breeds and mixed breeds in the same rule. Here’s a sample ruleset with two rules:

PUT _query_rules/dog-breed-ruleset
{
 "ruleset_id": "dog-breed-ruleset",
 "rules": [
   {
     "rule_id": "pug-mixes",
     "type": "pinned",
     "criteria": [
       {
         "type": "exact",
         "metadata": "breed",
         "values": [
           "pug"
         ]
       }
     ],
     "actions": {
       "ids": [
         "pug",
         "puggle",
         "chug",
         "pugshire"
       ]
     },
     "priority": 5
   },
   {
     "rule_id": "chi-mixes",
     "type": "pinned",
     "criteria": [
       {
         "type": "exact",
         "metadata": "breed",
         "values": [
           "chihauhua"
         ]
       }
     ],
     "actions": {
       "ids": [
         "chihauhua",
         "chiweenie",
         "chug"
       ]
     },
     "priority": 10
   }
 ]
}

Unpacking what this ruleset does:

  1. If the rule query sends in breed: pug, the first results in order will be: pug, puggle, chug, and pugshire. Any organic results will be returned after those pinned results.
  2. If the rule query sends in breed: chihuahua, the first results in order will be: chihuahua, chiweenie, and chug. Any organic results will be returned after those pinned results.

One thing to note is the priority of each rule in the ruleset. This is an optional part of each rule, but it can help you define the position in which an individual rule is inserted into the ruleset. Rulesets are sorted in ascending priority order, though as you can see in this example they don’t have to be contiguous. Inserting a new rule into this ruleset with a priority of 4 or lower would insert the new rule at the beginning of the ruleset, while a priority between 6 and 9 would insert the new rule in the middle of the ruleset between the two existing rules. If an individual rule is added to the ruleset but doesn’t include a priority, it will simply be appended to the end of the ruleset.

Another useful feature of query rules is that we now support multiple rulesets to be passed into the rule query. This allows you to organize and define many more rules; previously you were limited to the number of rules that could fit into a single ruleset (100 by default or configurable up to 1000 per ruleset).

Let’s take a look at what this looks like, by creating a new ruleset specifically for a July promotion. This rule is of type “always” so it will always be returned no matter what:

PUT _query_rules/promo-ruleset
{
  "rules": [
    {
      "rule_id": "july-promo",
      "type": "pinned",
      "criteria": [
        {
          "type": "always"
        }
      ],
      "actions": {
        "ids": [
          "july-promotion"
        ]
      }
    }
  ]
}

Now, we can query these rulesets together using the following query:

GET query-rules-test/_search
{
  "query": {
    "rule": {
      "organic": {
        "query_string": {
          "query": "chihauhua mixes"
        }
      },
      "ruleset_ids": [
        "promo-ruleset",
        "dog-breed-ruleset"
      ],
      "match_criteria": {
        "breed": "chihauhua"
      }
    }
  }
}

Since two rulesets were specified, we’ll process each ruleset in the order they were specified in the request. This means that the July promotion rule will always be returned as the very first result, and the matching dog breeds will be pinned subsequently, in a result that looks like this:

   "hits": [
     {
       "_index": "query-rules-test",
       "_id": "july-promotion",
       "_score": 1.7014128e+38,
       "_source": {
         "advert": "EVERYTHING ON SALE!"
       }
     },
     {
       "_index": "query-rules-test",
       "_id": "chihauhua",
       "_score": 1.7014126e+38,
       "_source": {
         "dog_breed": "chihauhua"
       }
     },
     {
       "_index": "query-rules-test",
       "_id": "chiweenie",
       "_score": 1.7014124e+38,
       "_source": {
         "dog_breed": "chiweenie"
       }
     },
     {
       "_index": "query-rules-test",
       "_id": "chug",
       "_score": 1.7014122e+38,
       "_source": {
         "dog_breed": "chug"
       }
     },
     ... // organic results follow... 
   ]

Complex query rule examples

Let’s continue our dog theme, but expand the mappings in our test index:

PUT query-rules-test
{
 "mappings": {
   "properties": {
     "breed": {
       "type": "keyword"
     },
     "age": {
       "type": "integer"
     },
     "sex": {
       "type": "keyword"
     },
     "name": {
       "type": "keyword"
     },
     "bio": {
       "type": "text"
     },
     "good_with_animals": {
       "type": "boolean"
     },
     "good_with_kids": {
        "type": "boolean"
     }
   }
 }
}

We can index the following json into this index for some sample data. For consistency, we're going to assume the _id field matches the json id field:

[
  {"id":"buddy_pug","breed":"pug","sex":"Male","age":3,"name":"Buddy","bio":"Buddy is a charming pug who loves to play and snuggle. He is looking for a loving home.","good_with_animals":true,"good_with_kids":true},
  {"id":"lucy_beagle","breed":"beagle","sex":"Female","age":7,"name":"Lucy","bio":"Lucy is a friendly beagle who enjoys long walks and is very affectionate.","good_with_animals":true,"good_with_kids":true},
  {"id":"rocky_chihuahua","breed":"chihuahua","sex":"Male","age":2,"name":"Rocky","bio":"Rocky is a tiny chihuahua with a big personality. He is very playful and loves attention.","good_with_animals":false,"good_with_kids":false},
  {"id":"zoe_dachshund","breed":"dachshund","sex":"Female","age":5,"name":"Zoe","bio":"Zoe is a sweet dachshund who loves to burrow under blankets and is very loyal.","good_with_animals":true,"good_with_kids":true},
  {"id":"max_poodle","breed":"poodle","sex":"Male","age":6,"name":"Max","bio":"Max is a smart and active poodle who loves learning new tricks and is very friendly.","good_with_animals":true,"good_with_kids":true},
  {"id":"bella_yorkie","breed":"yorkie","sex":"Female","age":4,"name":"Bella","bio":"Bella is a cute yorkie who loves to be pampered and is very affectionate.","good_with_animals":true,"good_with_kids":true},
  {"id":"jack_puggle","breed":"puggle","sex":"Male","age":8,"name":"Jack","bio":"Jack is a friendly puggle who loves to play and is very social.","good_with_animals":true,"good_with_kids":true},
  {"id":"lola_chiweenie","breed":"chiweenie","sex":"Female","age":9,"name":"Lola","bio":"Lola is a playful chiweenie who loves to chase toys and is very affectionate.","good_with_animals":true,"good_with_kids":true},
  {"id":"charlie_chug","breed":"chug","sex":"Male","age":3,"name":"Charlie","bio":"Charlie is an energetic chug who loves to play and is very curious.","good_with_animals":false,"good_with_kids":true},
  {"id":"daisy_pugshire","breed":"pugshire","sex":"Female","age":7,"name":"Daisy","bio":"Daisy is a gentle pugshire who loves to cuddle and is very sweet.","good_with_animals":true,"good_with_kids":true},
  {"id":"buster_pug","breed":"pug","sex":"Male","age":10,"name":"Buster","bio":"Buster is a calm pug who loves to lounge around and is very loyal.","good_with_animals":true,"good_with_kids":true},
  {"id":"molly_beagle","breed":"beagle","sex":"Female","age":12,"name":"Molly","bio":"Molly is a sweet beagle who loves to sniff around and is very friendly.","good_with_animals":true,"good_with_kids":true},
  {"id":"toby_chihuahua","breed":"chihuahua","sex":"Male","age":4,"name":"Toby","bio":"Toby is a lively chihuahua who loves to run around and is very playful.","good_with_animals":false,"good_with_kids":false},
  {"id":"luna_dachshund","breed":"dachshund","sex":"Female","age":1,"name":"Luna","bio":"Luna is a young dachshund who loves to explore and is very curious.","good_with_animals":true,"good_with_kids":true},
  {"id":"oliver_poodle","breed":"poodle","sex":"Male","age":17,"name":"Oliver","bio":"Oliver is a wise poodle who loves to relax and is very gentle.","good_with_animals":true,"good_with_kids":true}]

Next, let's create a ruleset:

PUT _query_rules/rescue-dog-search-ruleset
{
  "rules": [
    {
      "rule_id": "pugs_and_pug_mixes",
      "type": "pinned",
      "criteria": [
        {
          "type": "contains",
          "metadata": "query_string",
          "values": [
            "pug"
          ]
        }
      ],
      "actions": {
        "ids": [
          "buddy_pug",
          "buster_pug",
          "jack_puggle",
          "daisy_pugshire"
        ]
      }
    },
    {
      "rule_id": "puppies",
      "type": "pinned",
      "criteria": [
        {
          "type": "contains",
          "metadata": "query_string",
          "values": [
            "puppy",
            "puppies"
          ]
        }
      ],
      "actions": {
        "ids": [
          "luna_dachsund"
        ]
      }
    },
    {
      "rule_id": "puppies2",
      "type": "pinned",
      "criteria": [
        {
          "type": "lte",
          "metadata": "preferred_age",
          "values": [
            2
          ]
        }
      ],
      "actions": {
        "ids": [
          "luna_dachshund"
        ]
      }
    },
    {
      "rule_id": "adult",
      "type": "pinned",
      "criteria": [
        {
          "type": "gt",
          "metadata": "preferred_age",
          "values": [
            2
          ]
        },
        {
          "type": "lt",
          "metadata": "preferred_age",
          "values": [
            10
          ]
        }
      ],
      "actions": {
        "ids": [
          "lucy_beagle"
        ]
      }
    },
    {
      "rule_id": "special_needs",
      "type": "pinned",
      "criteria": [
        {
          "type": "exact",
          "metadata": "work_from_home",
          "values": [
            "true"
          ]
        },
        {
          "type": "exact",
          "metadata": "fenced_in_yard",
          "values": [
            "true"
          ]
        },
        {
          "type": "exact",
          "metadata": "kids_in_house",
          "values": [
            "false"
          ]
        },
        {
          "type": "exact",
          "metadata": "cats_in_house",
          "values": [
            "false"
          ]
        }
      ],
      "actions": {
        "ids": [
          "toby_chihuahua"
        ]
      }
    }
  ]
}

Unpacking this ruleset:

  • There’s a pugs_and_pug_mixes rule, which will pin all pugs and pug mixes if the query string contains the word “pug”
  • The puppies rule will return the youngest dog in our index if the query string contains “puppy” or “puppies”
  • Another puppies2 rule will return the same dog if the preferred age is less than or equal to 2
  • The adult rule will pin a specific beagle if the preferred age is between 2 and 10
  • We have a complex special_needs rule that will only hit if the prospective owner works from home, has a fenced in yard, and has no children or cats living in the house.

Let’s look at some example queries - these examples use match_none as the organic query, so the only results that are returned are for the rules themselves.

GET query-rules-test/_search
{
  "query": {
    "rule": {
      "organic": {
        "match_none": {}
      },
      "ruleset_ids": [
        "rescue-dog-search-ruleset"
      ],
      "match_criteria": {
        "query_string": "I like pugs and pug mixes",
        "preferred_age": 5
      }
    }
  }
}

The above query will match the “pug and pug mixes” rule and return the 4 pug results. However, since the preferred age is 5, we will return the adult dog, Lucy the beagle, as the 5th pinned result.

Changing the preferred age to 1 swaps the 5th results below the pug results for the dachshund puppy:

GET query-rules-test/_search
{
  "query": {
    "rule": {
      "organic": {
        "match_none": {}
      },
      "ruleset_ids": [
        "rescue-dog-search-ruleset"
      ],
      "match_criteria": {
        "query_string": "I like pugs and pug mixes",
        "preferred_age": 1
      }
    }
  }
}

In order to match the special needs result, all of the criteria must match:

{
  "query": {
    "rule": {
      "organic": {
        "match_none": {}
      },
      "ruleset_ids": [
        "rescue-dog-search-ruleset"
      ],
      "match_criteria": {
        "work_from_home": "true",
        "fenced_in_yard": "true",
        "kids_in_house": "false",
        "cats_in_house": "false"
      }
    }
  }
}

If a single criteria does not match, this rule will not be triggered:

GET query-rules-test/_search
{
  "query": {
    "rule": {
      "organic": {
        "match_none": {}
      },
      "ruleset_ids": [
        "rescue-dog-search-ruleset"
      ],
      "match_criteria": {
        "work_from_home": "true",
        "fenced_in_yard": "true",
        "kids_in_house": "false",
        "cats_in_house": "true"
      }
    }
  }
}

Troubleshooting query rules

It may be hard to tell whether a pinned query rule was applied. There are two ways to tell whether query rules are pinning the document that you want vs. being returned organically.

We do have the explain API, but this can be non-obvious: rule queries are rewritten into pinned queries which are then rewritten to constant score queries, so this will basically look like the max possible score:

"_explanation": {
  "value": 1.7014128e+38,
  "description": "max of:",
  "details": [
    {
      "value": 1.7014128e+38,
      "description": "max of:",
      "details": [
        {
          "value": 1.7014128e+38,
          "description": "ConstantScore(_id:([ff 63 68 69 68 61 75 68 75 61]))^1.7014128E38",
          "details": []
        },
        ...
      ]
    },
    ...
  ]
}

For pinned rules, you can also return only the rule by using a rule query combined with a match none query, as we did above for illustrative purposes:

GET query-rules-test/_search
{
  "query": {
    "rule": {
      "organic": {
        "match_none": {}
      },
      "ruleset_ids": [
        "dog-breed-ruleset",
        "super-special-ruleset"
      ],
      "match_criteria": {
        "breed": "pug"
      }
    }
  }
}

Alternately you can simply run the organic query by itself and compare the results of that query with your rule query.

What's next?

Just because query rules has been made generally available doesn’t mean that we are done!

We have a lot of exciting features on our roadmap, including:

  • Adding support for excluded documents as well as pinned documents
  • A UI to easily manage query rules without making API calls
  • Analyzer support
  • ... and more!

We’re also looking at other ways people may be interested in using query rules. We’d love to hear your feedback on our community page!

Ready to try this out on your own? Start a free trial.

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!

Related content

Interval queries: why they are true positional queries, and how to transition from Span
Search RelevanceHow To

November 1, 2024

Interval queries: why they are true positional queries, and how to transition from Span

Explains how Interval queries are true positional queries and how to transition to them from Span queries.

MS
By: Mayya Sharipova
Personalized search with LTR
Search Relevance

August 30, 2024

Personalized search with LTR

Learn how to train ranking models that improve search relevance for individual users.

MJ
By: Max Jakob
Looking back: A timeline of vector search innovations
Vector DatabaseSearch Relevance

August 27, 2024

Looking back: A timeline of vector search innovations

Looking back at Elastic's vector search innovations in Elasticsearch and Lucene

KDBT
By: Kathleen DeRusso and Benjamin Trent
Introducing Learning To Rank (LTR) in Elasticsearch
Search RelevanceHow To

July 15, 2024

Introducing Learning To Rank (LTR) in Elasticsearch

Discover how Learning To Rank (LTR) can help you to improve your search ranking and how to implement it in Elasticsearch.

AF
By: Aurélien Foucret
Semantic reranking in Elasticsearch with retrievers
Vector DatabaseSearch Relevance+1

May 28, 2024

Semantic reranking in Elasticsearch with retrievers

Explore strategies for using semantic reranking to boost the relevance of top search results, including semantic reranking with retrievers.

ADNC
By: Adam Demjen and Nick Chow

Ready to build state of the art search experiences?

Sufficiently advanced search isn’t achieved with the efforts of one. Elasticsearch is powered by data scientists, ML ops, engineers, and many more who are just as passionate about search as your are. Let’s connect and work together to build the magical search experience that will get you the results you want.

Try it yourself