New

The executive guide to generative AI

Read more
About usPartnersSupport|Login

WARNING: Version 5.1 of Elasticsearch has passed its EOL date.

This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.

« Stop Token Filter Stemmer Token Filter »
Elastic Docs ›Elasticsearch Guide [5.1] ›Analysis ›Token Filters

Word Delimiter Token Filter

edit

Word Delimiter Token Filter

edit

Named word_delimiter, it Splits words into subwords and performs optional transformations on subword groups. Words are split into subwords with the following rules:

  • split on intra-word delimiters (by default, all non alpha-numeric characters).
  • "Wi-Fi" → "Wi", "Fi"
  • split on case transitions: "PowerShot" → "Power", "Shot"
  • split on letter-number transitions: "SD500" → "SD", "500"
  • leading and trailing intra-word delimiters on each subword are ignored: "//hello---there, dude" → "hello", "there", "dude"
  • trailing "'s" are removed for each subword: "O’Neil’s" → "O", "Neil"

Parameters include:

generate_word_parts
If true causes parts of words to be generated: "PowerShot" ⇒ "Power" "Shot". Defaults to true.
generate_number_parts
If true causes number subwords to be generated: "500-42" ⇒ "500" "42". Defaults to true.
catenate_words
If true causes maximum runs of word parts to be catenated: "wi-fi" ⇒ "wifi". Defaults to false.
catenate_numbers
If true causes maximum runs of number parts to be catenated: "500-42" ⇒ "50042". Defaults to false.
catenate_all
If true causes all subword parts to be catenated: "wi-fi-4000" ⇒ "wifi4000". Defaults to false.
split_on_case_change
If true causes "PowerShot" to be two tokens; ("Power-Shot" remains two parts regards). Defaults to true.
preserve_original
If true includes original words in subwords: "500-42" ⇒ "500-42" "500" "42". Defaults to false.
split_on_numerics
If true causes "j2se" to be three tokens; "j" "2" "se". Defaults to true.
stem_english_possessive
If true causes trailing "'s" to be removed for each subword: "O’Neil’s" ⇒ "O", "Neil". Defaults to true.

Advance settings include:

protected_words
A list of protected words from being delimiter. Either an array, or also can set protected_words_path which resolved to a file configured with protected words (one on each line). Automatically resolves to config/ based location if exists.
type_table
A custom type mapping table, for example (when configured using type_table_path): 
    # Map the $, %, '.', and ',' characters to DIGIT
    # This might be useful for financial data.
    $ => DIGIT
    % => DIGIT
    . => DIGIT
    \\u002C => DIGIT

    # in some cases you might not want to split on ZWJ
    # this also tests the case where we need a bigger byte[]
    # see http://en.wikipedia.org/wiki/Zero-width_joiner
    \\u200D => ALPHANUM

Using a tokenizer like the standard tokenizer may interfere with the catenate_* and preserve_original parameters, as the original string may already have lost punctuation during tokenization. Instead, you may want to use the whitespace tokenizer.

« Stop Token Filter Stemmer Token Filter »
Was this helpful?
Feedback