New

The executive guide to generative AI

Read more

Lowercase Tokenizer

edit

Lowercase Tokenizer

edit

A tokenizer of type lowercase that performs the function of Letter Tokenizer and Lower Case Token Filter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of Letter Tokenizer and Lower Case Token Filter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.

Was this helpful?
Feedback