Path Hierarchy Tokenizer

edit

The path_hierarchy tokenizer takes something like this:

/something/something/else

And produces tokens:

/something
/something/something
/something/something/else
Setting Description

delimiter

The character delimiter to use, defaults to /.

replacement

An optional replacement character to use. Defaults to the delimiter.

buffer_size

The buffer size to use, defaults to 1024.

reverse

Generates tokens in reverse order, defaults to false.

skip

Controls initial tokens to skip, defaults to 0.

type

Required. Must be set to PathHierarchy (case-sensitive).