Synonym graph token filter

The synonym_graph token filter allows to easily handle synonyms, including multi-word synonyms correctly during the analysis process.

In order to properly handle multi-word synonyms this token filter creates a graph token stream during processing. For more information on this topic and its various complexities, please read the Lucene’s TokenStreams are actually graphs blog post.

Synonyms in a synonyms set are defined using synonym rules. Each synonym rule contains words that are synonyms.

You can use two formats to define synonym rules: Solr and WordNet.

This format uses two different definitions:

WordNet defines synonyms sets spanning multiple lines. Each line contains the following information:

The following example defines a synonym set for the words "come", "advance" and "approach":

s(100000002,1,'come',v,1,0).
s(100000002,2,'advance',v,1,0).
s(100000002,3,'approach',v,1,0).""";

Synonyms can be configured using the synonyms API, a synonyms file, or directly inlined in the token filter configuration. See store your synonyms set for more details on each option.

Use synonyms_set configuration option to provide a synonym set created via Synonyms Management APIs:

  "filter": {
    "synonyms_filter": {
      "type": "synonym_graph",
      "synonyms_set": "my-synonym-set",
      "updateable": true
    }
  }

Use synonyms_path to provide a synonym file :

  "filter": {
    "synonyms_filter": {
      "type": "synonym_graph",
      "synonyms_path": "analysis/synonym-set.txt"
    }
  }

The above configures a synonym filter, with a path of analysis/synonym-set.txt (relative to the config location).

Use synonyms to define inline synonyms:

  "filter": {
    "synonyms_filter": {
      "type": "synonym_graph",
      "synonyms": ["pc => personal computer", "computer, pc, laptop"]
    }
  }

Additional settings are:

The expand parameter controls whether to expand equivalent synonym rules. Consider a synonym defined like:

foo, bar, baz

Using expand: true, the synonym rule would be expanded into:

foo => foo
foo => bar
foo => baz
bar => foo
bar => bar
bar => baz
baz => foo
baz => bar
baz => baz

When expand is set to false, the synonym rule is not expanded and the first synonym is treated as the canonical representation. The synonym would be equivalent to:

foo => foo
bar => foo
baz => foo

The expand parameter does not affect explicit synonym rules, like foo, bar => baz.

The tokenizer parameter controls the tokenizers that will be used to tokenize the synonym, this parameter is for backwards compatibility for indices that created before 6.0. The ignore_case parameter works with tokenizer parameter only.

To apply synonyms, you will need to include a synonym graph token filter into an analyzer:

      "analyzer": {
        "my_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["stemmer", "synonym_graph"]
        }
      }

Order is important for your token filters. Text will be processed first through filters preceding the synonym filter before being processed by the synonym filter.

Elasticsearch will also use the token filters preceding the synonym filter in a tokenizer chain to parse the entries in a synonym file or synonym set. In the above example, the synonyms graph token filter is placed after a stemmer. The stemmer will also be applied to the synonym entries.

Because entries in the synonym map cannot have stacked positions, some token filters may cause issues here. Token filters that produce multiple versions of a token may choose which version of the token to emit when parsing synonyms. For example, asciifolding will only produce the folded version of the token. Others, like multiplexer, word_delimiter_graph or ngram will throw an error.

If you need to build analyzers that include both multi-token filters and synonym filters, consider using the multiplexer filter, with the multi-token filters in one branch and the synonym filter in the other.

Synonyms and stop token filters interact with each other in the following ways:

Stop words will be removed from the synonym rule definition. This can can cause errors on the synonym rule.

For explicit synonym rules like foo, bar => baz with a stop filter that removes bar:

If the stop filter removed baz instead:

For equivalent synonym rules like foo, bar, baz and expand: true, with a stop filter that removes `bar:

foo => foo
foo => baz
baz => foo
baz => baz

The stop filter will remove the terms from the resulting synonym expansion.

For example, a synonym rule like foo, bar => baz and a stop filter that removes baz will get no matches for foo or bar, as both would get expanded to baz which is removed by the stop filter.

If the stop filter removed foo instead, then searching for foo would get expanded to baz, which is not removed by the stop filter thus potentially providing matches for baz.