nori_tokenizer
editnori_tokenizer
editThe nori_tokenizer
accepts the following settings:
-
decompound_mode
-
The decompound mode determines how the tokenizer handles compound tokens. It can be set to:
-
none
-
No decomposition for compounds. Example output:
가거도항 가곡역
-
discard
-
Decomposes compounds and discards the original form (default). Example output:
가곡역 => 가곡, 역
-
mixed
-
Decomposes compounds and keeps the original form. Example output:
가곡역 => 가곡역, 가곡, 역
-
-
discard_punctuation
-
Whether punctuation should be discarded from the output. Defaults to
true
. -
user_dictionary
-
The Nori tokenizer uses the mecab-ko-dic dictionary by default. A
user_dictionary
with custom nouns (NNG
) may be appended to the default dictionary. The dictionary should have the following format:<token> [<token 1> ... <token n>]
The first token is mandatory and represents the custom noun that should be added in the dictionary. For compound nouns the custom segmentation can be provided after the first token (
[<token 1> ... <token n>]
). The segmentation of the custom compound nouns is controlled by thedecompound_mode
setting.As a demonstration of how the user dictionary can be used, save the following dictionary to
$ES_HOME/config/userdict_ko.txt
:Then create an analyzer as follows:
PUT nori_sample { "settings": { "index": { "analysis": { "tokenizer": { "nori_user_dict": { "type": "nori_tokenizer", "decompound_mode": "mixed", "discard_punctuation": "false", "user_dictionary": "userdict_ko.txt" } }, "analyzer": { "my_analyzer": { "type": "custom", "tokenizer": "nori_user_dict" } } } } } } GET nori_sample/_analyze { "analyzer": "my_analyzer", "text": "세종시" }
The above
analyze
request returns the following:{ "tokens" : [ { "token" : "세종시", "start_offset" : 0, "end_offset" : 3, "type" : "word", "position" : 0, "positionLength" : 2 }, { "token" : "세종", "start_offset" : 0, "end_offset" : 2, "type" : "word", "position" : 0 }, { "token" : "시", "start_offset" : 2, "end_offset" : 3, "type" : "word", "position" : 1 }] }
-
user_dictionary_rules
-
You can also inline the rules directly in the tokenizer definition using the
user_dictionary_rules
option:PUT nori_sample { "settings": { "index": { "analysis": { "tokenizer": { "nori_user_dict": { "type": "nori_tokenizer", "decompound_mode": "mixed", "user_dictionary_rules": ["c++", "C쁠쁠", "세종", "세종시 세종 시"] } }, "analyzer": { "my_analyzer": { "type": "custom", "tokenizer": "nori_user_dict" } } } } } }
The nori_tokenizer
sets a number of additional attributes per token that are used by token filters
to modify the stream.
You can view all these additional attributes with the following request:
GET _analyze { "tokenizer": "nori_tokenizer", "text": "뿌리가 깊은 나무는", "attributes" : ["posType", "leftPOS", "rightPOS", "morphemes", "reading"], "explain": true }
Which responds with:
{ "detail": { "custom_analyzer": true, "charfilters": [], "tokenizer": { "name": "nori_tokenizer", "tokens": [ { "token": "뿌리", "start_offset": 0, "end_offset": 2, "type": "word", "position": 0, "leftPOS": "NNG(General Noun)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "NNG(General Noun)" }, { "token": "가", "start_offset": 2, "end_offset": 3, "type": "word", "position": 1, "leftPOS": "J(Ending Particle)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "J(Ending Particle)" }, { "token": "깊", "start_offset": 4, "end_offset": 5, "type": "word", "position": 2, "leftPOS": "VA(Adjective)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "VA(Adjective)" }, { "token": "은", "start_offset": 5, "end_offset": 6, "type": "word", "position": 3, "leftPOS": "E(Verbal endings)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "E(Verbal endings)" }, { "token": "나무", "start_offset": 7, "end_offset": 9, "type": "word", "position": 4, "leftPOS": "NNG(General Noun)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "NNG(General Noun)" }, { "token": "는", "start_offset": 9, "end_offset": 10, "type": "word", "position": 5, "leftPOS": "J(Ending Particle)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "J(Ending Particle)" } ] }, "tokenfilters": [] } }