Synonym Graph Token Filter
editSynonym Graph Token Filter
editThe synonym_graph
token filter allows to easily handle synonyms,
including multi-word synonyms correctly during the analysis process.
In order to properly handle multi-word synonyms this token filter creates a "graph token stream" during processing. For more information on this topic and its various complexities, please read the Lucene’s TokenStreams are actually graphs blog post.
This token filter is designed to be used as part of a search analyzer only. If you want to apply synonyms during indexing please use the standard synonym token filter.
Synonyms are configured using a configuration file. Here is an example:
PUT /test_index { "settings": { "index" : { "analysis" : { "analyzer" : { "search_synonyms" : { "tokenizer" : "whitespace", "filter" : ["graph_synonyms"] } }, "filter" : { "graph_synonyms" : { "type" : "synonym_graph", "synonyms_path" : "analysis/synonym.txt" } } } } } }
The above configures a search_synonyms
filter, with a path of
analysis/synonym.txt
(relative to the config
location). The
search_synonyms
analyzer is then configured with the filter.
Additional settings are:
-
expand
(defaults totrue
). -
lenient
(defaults tofalse
). Iftrue
ignores exceptions while parsing the synonym configuration. It is important to note that only those synonym rules which cannot get parsed are ignored. For instance consider the following request:
PUT /test_index { "settings": { "index" : { "analysis" : { "analyzer" : { "synonym" : { "tokenizer" : "standard", "filter" : ["my_stop", "synonym_graph"] } }, "filter" : { "my_stop": { "type" : "stop", "stopwords": ["bar"] }, "synonym_graph" : { "type" : "synonym_graph", "lenient": true, "synonyms" : ["foo, bar => baz"] } } } } } }
With the above request the word bar
gets skipped but a mapping foo => baz
is still added. However, if the mapping
being added was "foo, baz ⇒ bar" nothing would get added to the synonym list. This is because the target word for the
mapping is itself eliminated because it was a stop word. Similarly, if the mapping was "bar, foo, baz" and expand
was
set to false
no mapping would get added as when expand=false
the target mapping is the first word. However, if
expand=true
then the mappings added would be equivalent to foo, baz => foo, baz
i.e, all mappings other than the
stop word.
tokenizer
and ignore_case
are deprecated
editThe tokenizer
parameter controls the tokenizers that will be used to
tokenize the synonym, this parameter is for backwards compatibility for indices that created before 6.0..
The ignore_case
parameter works with tokenizer
parameter only.
Two synonym formats are supported: Solr, WordNet.
Solr synonyms
editThe following is a sample format of the file:
# Blank lines and lines starting with pound are comments. # Explicit mappings match any token sequence on the LHS of "=>" # and replace with all alternatives on the RHS. These types of mappings # ignore the expand parameter in the schema. # Examples: i-pod, i pod => ipod, sea biscuit, sea biscit => seabiscuit # Equivalent synonyms may be separated with commas and give # no explicit mapping. In this case the mapping behavior will # be taken from the expand parameter in the schema. This allows # the same synonym file to be used in different synonym handling strategies. # Examples: ipod, i-pod, i pod foozball , foosball universe , cosmos lol, laughing out loud # If expand==true, "ipod, i-pod, i pod" is equivalent # to the explicit mapping: ipod, i-pod, i pod => ipod, i-pod, i pod # If expand==false, "ipod, i-pod, i pod" is equivalent # to the explicit mapping: ipod, i-pod, i pod => ipod # Multiple synonym mapping entries are merged. foo => foo bar foo => baz # is equivalent to foo => foo bar, baz
You can also define synonyms for the filter directly in the
configuration file (note use of synonyms
instead of synonyms_path
):
PUT /test_index { "settings": { "index" : { "analysis" : { "filter" : { "synonym" : { "type" : "synonym_graph", "synonyms" : [ "lol, laughing out loud", "universe, cosmos" ] } } } } } }
However, it is recommended to define large synonyms set in a file using
synonyms_path
, because specifying them inline increases cluster size unnecessarily.
WordNet synonyms
editSynonyms based on WordNet format can be
declared using format
:
PUT /test_index { "settings": { "index" : { "analysis" : { "filter" : { "synonym" : { "type" : "synonym_graph", "format" : "wordnet", "synonyms" : [ "s(100000001,1,'abstain',v,1,0).", "s(100000001,2,'refrain',v,1,0).", "s(100000001,3,'desist',v,1,0)." ] } } } } } }
Using synonyms_path
to define WordNet synonyms in a file is supported
as well.
Parsing synonym files
editElasticsearch will use the token filters preceding the synonym filter
in a tokenizer chain to parse the entries in a synonym file. So, for example, if a
synonym filter is placed after a stemmer, then the stemmer will also be applied
to the synonym entries. Because entries in the synonym map cannot have stacked
positions, some token filters may cause issues here. Token filters that produce
multiple versions of a token may choose which version of the token to emit when
parsing synonyms, e.g. asciifolding
will only produce the folded version of the
token. Others, e.g. multiplexer
, word_delimiter_graph
or ngram
will throw an
error.
The synonym rules should not contain words that are removed by
a filter that appears after in the chain (a stop
filter for instance).
Removing a term from a synonym rule breaks the matching at query time.