Stopwords and Performance

The easiest way to reduce the number of documents is simply to use the and operator with the match query, in order to make all words required.

A match query like this:

{
    "match": {
        "text": {
            "query":    "the quick brown fox",
            "operator": "and"
        }
    }
}

is rewritten as a bool query like this:

{
    "bool": {
        "must": [
            { "term": { "text": "the" }},
            { "term": { "text": "quick" }},
            { "term": { "text": "brown" }},
            { "term": { "text": "fox" }}
        ]
    }
}

The bool query is intelligent enough to execute each term query in the optimal order—​it starts with the least frequent term. Because all terms are required, only documents that contain the least frequent term can possibly match. Using the and operator greatly speeds up multiterm queries.

In Controlling Precision, we discussed using the minimum_should_match operator to trim the long tail of less-relevant results. It is useful for this purpose alone but, as a nice side effect, it offers a similar performance benefit to the and operator:

{
    "match": {
        "text": {
            "query": "the quick brown fox",
            "minimum_should_match": "75%"
        }
    }
}

In this example, at least three out of the four terms must match. This means that the only docs that need to be considered are those that contain either the least or second least frequent terms.

This offers a huge performance gain over a simple query with the default or operator! But we can do better yet…​