Compound Word Token Filters
editCompound Word Token Filters
editThe hyphenation_decompounder
and dictionary_decompounder
token filters can
decompose compound words found in many German languages into word parts.
Both token filters require a dictionary of word parts, which can be provided as:
|
An array of words, specified inline in the token filter configuration, or |
|
The path (either absolute or relative to the |
Hyphenation decompounder
editThe hyphenation_decompounder
uses hyphenation grammars to find potential
subwords that are then checked against the word dictionary. The quality of the
output tokens is directly connected to the quality of the grammar file you
use. For languages like German they are quite good.
XML based hyphenation grammar files can be found in the
Objects For Formatting Objects
(OFFO) Sourceforge project. Currently only FOP v1.2 compatible hyphenation files
are supported. You can download offo-hyphenation_v1.2.zip
directly and look in the offo-hyphenation/hyph/
directory.
Credits for the hyphenation code go to the Apache FOP project .
Dictionary decompounder
editThe dictionary_decompounder
uses a brute force approach in conjunction with
only the word dictionary to find subwords in a compound word. It is much
slower than the hyphenation decompounder but can be used as a first start to
check the quality of your dictionary.
Compound token filter parameters
editThe following parameters can be used to configure a compound word token filter:
|
Either |
|
A array containing a list of words to use for the word dictionary. |
|
The path (either absolute or relative to the |
|
The path (either absolute or relative to the |
|
Minimum word size. Defaults to 5. |
|
Minimum subword size. Defaults to 2. |
|
Maximum subword size. Defaults to 15. |
|
Whether to include only the longest matching subword or not. Defaults to |
Here is an example:
PUT /compound_word_example { "settings": { "index": { "analysis": { "analyzer": { "my_analyzer": { "type": "custom", "tokenizer": "standard", "filter": ["dictionary_decompounder", "hyphenation_decompounder"] } }, "filter": { "dictionary_decompounder": { "type": "dictionary_decompounder", "word_list": ["one", "two", "three"] }, "hyphenation_decompounder": { "type" : "hyphenation_decompounder", "word_list_path": "analysis/example_word_list.txt", "hyphenation_patterns_path": "analysis/hyphenation_patterns.xml", "max_subword_size": 22 } } } } } }