Analysis

Analysis is the process of converting text, like the body of any email, into tokens or terms which are added to the inverted index for searching. Analysis is performed by an analyzer which can be either a built-in analyzer or a custom analyzer defined per index.

For instance, at index time the built-in english analyzer will first convert the sentence:

"The QUICK brown foxes jumped over the lazy dog!"

into distinct tokens. It will then lowercase each token, remove frequent stopwords ("the") and reduce the terms to their word stems (foxes → fox, jumped → jump, lazy → lazi). In the end, the following terms will be added to the inverted index:

[ quick, brown, fox, jump, over, lazi, dog ]

Each text field in a mapping can specify its own analyzer:

PUT my_index
{
  "mappings": {
    "properties": {
      "title": {
        "type":     "text",
        "analyzer": "standard"
      }
    }
  }
}

At index time, if no analyzer has been specified, it looks for an analyzer in the index settings called default. Failing that, it defaults to using the standard analyzer.

This same analysis process is applied to the query string at search time in full text queries like the match query to convert the text in the query string into terms of the same form as those that are stored in the inverted index.

For instance, a user might search for:

"a quick fox"

which would be analysed by the same english analyzer into the following terms:

[ quick, fox ]

Even though the exact words used in the query string don’t appear in the original text (quick vs QUICK, fox vs foxes), because we have applied the same analyzer to both the text and the query string, the terms from the query string exactly match the terms from the text in the inverted index, which means that this query would match our example document.

Usually the same analyzer should be used both at index time and at search time, and full text queries like the match query will use the mapping to look up the analyzer to use for each field.

The analyzer to use to search a particular field is determined by looking for: