| Class | Description |
|---|---|
| CommonCornerCasesTest | |
| TestUtils | |
| TokenBase |
Abstract token class with features shared by all tokens produced by all tokenizers
|
| TokenizerBase |
TokenizerBase main class
|
| TokenizerBase.Builder |
Abstract Builder shared by all tokenizers
|
| Enum | Description |
|---|---|
| TokenizerBase.Mode |