WARNING: Version 2.0 of Elasticsearch has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
ICU Analysis Plugin
editICU Analysis Plugin
editThe ICU analysis plugin allows for unicode normalization, collation and folding. The plugin is called elasticsearch-analysis-icu.
The plugin includes the following analysis components:
ICU Normalization
editNormalizes characters as explained
here. It
registers itself by default under icu_normalizer
or icuNormalizer
using the default settings. Allows for the name parameter to be provided
which can include the following values: nfc
, nfkc
, and nfkc_cf
.
Here is a sample settings:
{ "index" : { "analysis" : { "analyzer" : { "normalization" : { "tokenizer" : "keyword", "filter" : ["icu_normalizer"] } } } } }
ICU Folding
editFolding of unicode characters based on UTR#30
. It registers itself
under icu_folding
and icuFolding
names.
The filter also does lowercasing, which means the lowercase filter can
normally be left out. Sample setting:
{ "index" : { "analysis" : { "analyzer" : { "folding" : { "tokenizer" : "keyword", "filter" : ["icu_folding"] } } } } }
Filtering
editThe folding can be filtered by a set of unicode characters with the
parameter unicodeSetFilter
. This is useful for a non-internationalized
search engine where retaining a set of national characters which are
primary letters in a specific language is wanted. See syntax for the
UnicodeSet
here.
The Following example exempts Swedish characters from the folding. Note that the filtered characters are NOT lowercased which is why we add that filter below.
{ "index" : { "analysis" : { "analyzer" : { "folding" : { "tokenizer" : "standard", "filter" : ["my_icu_folding", "lowercase"] } } "filter" : { "my_icu_folding" : { "type" : "icu_folding" "unicodeSetFilter" : "[^åäöÅÄÖ]" } } } } }
ICU Collation
editUses collation token filter. Allows to either specify the rules for
collation (defined
here)
using the rules
parameter (can point to a location or expressed in the
settings, location can be relative to config location), or using the
language
parameter (further specialized by country and variant). By
default registers under icu_collation
or icuCollation
and uses the
default locale.
Here is a sample settings:
{ "index" : { "analysis" : { "analyzer" : { "collation" : { "tokenizer" : "keyword", "filter" : ["icu_collation"] } } } } }
And here is a sample of custom collation:
{ "index" : { "analysis" : { "analyzer" : { "collation" : { "tokenizer" : "keyword", "filter" : ["myCollator"] } }, "filter" : { "myCollator" : { "type" : "icu_collation", "language" : "en" } } } } }
Options
edit
|
The strength property determines the minimum level of difference considered significant during comparison. The default strength for the Collator is See ICU Collation documentation for a more detailed explanation for the specific values. |
|
Possible values: |
Expert options:
edit
|
Possible values: |
|
Possible values: |
|
Possible values: |
|
Possible values: |
|
Single character or contraction. Controls what is variable for |
|
Possible values: |
ICU Tokenizer
editBreaks text into words according to UAX #29: Unicode Text Segmentation http://www.unicode.org/reports/tr29/.
{ "index" : { "analysis" : { "analyzer" : { "collation" : { "tokenizer" : "icu_tokenizer", } } } } }
ICU Normalization CharFilter
editNormalizes characters as explained here.
It registers itself by default under icu_normalizer
or icuNormalizer
using the default settings.
Allows for the name parameter to be provided which can include the following values: nfc
, nfkc
, and nfkc_cf
.
Allows for the mode parameter to be provided which can include the following values: compose
and decompose
.
Use decompose
with nfc
or nfkc
, to get nfd
or nfkd
, respectively.
Here is a sample settings:
{ "index" : { "analysis" : { "analyzer" : { "collation" : { "tokenizer" : "keyword", "char_filter" : ["icu_normalizer"] } } } } }