IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Synonym Token Filter Compound Word Token Filters »
Elastic Docs ›Elasticsearch Guide [6.5] ›Analysis ›Token Filters

Synonym Graph Token Filter

edit

Synonym Graph Token Filter

edit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

The synonym_graph token filter allows to easily handle synonyms, including multi-word synonyms correctly during the analysis process.

In order to properly handle multi-word synonyms this token filter creates a "graph token stream" during processing. For more information on this topic and its various complexities, please read the Lucene’s TokenStreams are actually graphs blog post.

This token filter is designed to be used as part of a search analyzer only. If you want to apply synonyms during indexing please use the standard synonym token filter.

Synonyms are configured using a configuration file. Here is an example:

PUT /test_index
{
    "settings": {
        "index" : {
            "analysis" : {
                "analyzer" : {
                    "search_synonyms" : {
                        "tokenizer" : "whitespace",
                        "filter" : ["graph_synonyms"]
                    }
                },
                "filter" : {
                    "graph_synonyms" : {
                        "type" : "synonym_graph",
                        "synonyms_path" : "analysis/synonym.txt"
                    }
                }
            }
        }
    }
}

The above configures a search_synonyms filter, with a path of analysis/synonym.txt (relative to the config location). The search_synonyms analyzer is then configured with the filter.

Additional settings are:

  • expand (defaults to true).
  • lenient (defaults to false). If true ignores exceptions while parsing the synonym configuration. It is important to note that only those synonym rules which cannot get parsed are ignored. For instance consider the following request: 
PUT /test_index
{
    "settings": {
        "index" : {
            "analysis" : {
                "analyzer" : {
                    "synonym" : {
                        "tokenizer" : "standard",
                        "filter" : ["my_stop", "synonym_graph"]
                    }
                },
                "filter" : {
                	"my_stop": {
                		"type" : "stop",
                		"stopwords": ["bar"]
                	},
                    "synonym_graph" : {
                        "type" : "synonym_graph",
                        "lenient": true,
                        "synonyms" : ["foo, bar => baz"]
                    }
                }
            }
        }
    }
}

With the above request the word bar gets skipped but a mapping foo => baz is still added. However, if the mapping being added was "foo, baz ⇒ bar" nothing would get added to the synonym list. This is because the target word for the mapping is itself eliminated because it was a stop word. Similarly, if the mapping was "bar, foo, baz" and expand was set to false no mapping would get added as when expand=false the target mapping is the first word. However, if expand=true then the mappings added would be equivalent to foo, baz => foo, baz i.e, all mappings other than the stop word.

tokenizer and ignore_case are deprecated

edit

The tokenizer parameter controls the tokenizers that will be used to tokenize the synonym, this parameter is for backwards compatibility for indices that created before 6.0.. The ignore_case parameter works with tokenizer parameter only.

Two synonym formats are supported: Solr, WordNet.

Solr synonyms

edit

The following is a sample format of the file:

# Blank lines and lines starting with pound are comments.

# Explicit mappings match any token sequence on the LHS of "=>"
# and replace with all alternatives on the RHS.  These types of mappings
# ignore the expand parameter in the schema.
# Examples:
i-pod, i pod => ipod,
sea biscuit, sea biscit => seabiscuit

# Equivalent synonyms may be separated with commas and give
# no explicit mapping.  In this case the mapping behavior will
# be taken from the expand parameter in the schema.  This allows
# the same synonym file to be used in different synonym handling strategies.
# Examples:
ipod, i-pod, i pod
foozball , foosball
universe , cosmos
lol, laughing out loud

# If expand==true, "ipod, i-pod, i pod" is equivalent
# to the explicit mapping:
ipod, i-pod, i pod => ipod, i-pod, i pod
# If expand==false, "ipod, i-pod, i pod" is equivalent
# to the explicit mapping:
ipod, i-pod, i pod => ipod

# Multiple synonym mapping entries are merged.
foo => foo bar
foo => baz
# is equivalent to
foo => foo bar, baz

You can also define synonyms for the filter directly in the configuration file (note use of synonyms instead of synonyms_path):

PUT /test_index
{
    "settings": {
        "index" : {
            "analysis" : {
                "filter" : {
                    "synonym" : {
                        "type" : "synonym_graph",
                        "synonyms" : [
                            "lol, laughing out loud",
                            "universe, cosmos"
                        ]
                    }
                }
            }
        }
    }
}

However, it is recommended to define large synonyms set in a file using synonyms_path, because specifying them inline increases cluster size unnecessarily.

WordNet synonyms

edit

Synonyms based on WordNet format can be declared using format:

PUT /test_index
{
    "settings": {
        "index" : {
            "analysis" : {
                "filter" : {
                    "synonym" : {
                        "type" : "synonym_graph",
                        "format" : "wordnet",
                        "synonyms" : [
                            "s(100000001,1,'abstain',v,1,0).",
                            "s(100000001,2,'refrain',v,1,0).",
                            "s(100000001,3,'desist',v,1,0)."
                        ]
                    }
                }
            }
        }
    }
}

Using synonyms_path to define WordNet synonyms in a file is supported as well.

« Synonym Token Filter Compound Word Token Filters »