Class | Description |
---|---|
CommonCornerCasesTest | |
TestUtils | |
TokenBase |
Abstract token class with features shared by all tokens produced by all tokenizers
|
TokenizerBase |
TokenizerBase main class
|
TokenizerBase.Builder |
Abstract Builder shared by all tokenizers
|
Enum | Description |
---|---|
TokenizerBase.Mode |