WARNING: Version 5.6 of Elasticsearch has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
Common Grams Token Filter
editCommon Grams Token Filter
editToken filter that generates bigrams for frequently occurring terms. Single terms are still indexed. It can be used as an alternative to the Stop Token Filter when we don’t want to completely ignore common terms.
For example, the text "the quick brown is a fox" will be tokenized as "the", "the_quick", "quick", "brown", "brown_is", "is_a", "a_fox", "fox". Assuming "the", "is" and "a" are common words.
When query_mode
is enabled, the token filter removes common words and
single terms followed by a common word. This parameter should be enabled
in the search analyzer.
For example, the query "the quick brown is a fox" will be tokenized as "the_quick", "quick", "brown_is", "is_a", "a_fox", "fox".
The following are settings that can be set:
Setting | Description |
---|---|
|
A list of common words to use. |
|
A path (either relative to |
|
If true, common words matching will be case insensitive
(defaults to |
|
Generates bigrams then removes common words and single
terms followed by a common word (defaults to |
Note, common_words
or common_words_path
field is required.
Here is an example:
PUT /common_grams_example { "settings": { "analysis": { "my_analyzer": { "index_grams": { "tokenizer": "whitespace", "filter": ["common_grams"] }, "search_grams": { "tokenizer": "whitespace", "filter": ["common_grams_query"] } }, "filter": { "common_grams": { "type": "common_grams", "common_words": ["a", "an", "the"] }, "common_grams_query": { "type": "common_grams", "query_mode": true, "common_words": ["a", "an", "the"] } } } } }