Simple Pattern Tokenizer
editSimple Pattern Tokenizer
editThis functionality is marked as experimental in Lucene
The simple_pattern
tokenizer uses a regular expression to capture matching
text as terms. The set of regular expression features it supports is more
limited than the pattern
tokenizer, but the
tokenization is generally faster.
This tokenizer does not support splitting the input on a pattern match, unlike
the pattern
tokenizer. To split on pattern
matches using the same restricted regular expression subset, see the
simple_pattern_split
tokenizer.
This tokenizer uses Lucene regular expressions. For an explanation of the supported features and syntax, see Regular Expression Syntax.
The default pattern is the empty string, which produces no terms. This tokenizer should always be configured with a non-default pattern.
Configuration
editThe simple_pattern
tokenizer accepts the following parameters:
|
Lucene regular expression, defaults to the empty string. |
Example configuration
editThis example configures the simple_pattern
tokenizer to produce terms that are
three-digit numbers
PUT my_index { "settings": { "analysis": { "analyzer": { "my_analyzer": { "tokenizer": "my_tokenizer" } }, "tokenizer": { "my_tokenizer": { "type": "simple_pattern", "pattern": "[0123456789]{3}" } } } } } POST my_index/_analyze { "analyzer": "my_analyzer", "text": "fd-786-335-514-x" }
The above example produces these terms:
[ 786, 335, 514 ]