public interface TokenizerFactory
Modifier and Type | Method and Description |
---|---|
Tokenizer |
create(java.io.InputStream toTokenize)
Create a tokenizer based on an input stream
|
Tokenizer |
create(java.lang.String toTokenize)
The tokenizer to createComplex
|
TokenPreProcess |
getTokenPreProcessor()
Returns TokenPreProcessor set for this TokenizerFactory instance
|
void |
setTokenPreProcessor(TokenPreProcess preProcessor)
Sets a token pre processor to be used
with every tokenizer
|
Tokenizer create(java.lang.String toTokenize)
toTokenize
- the string to createComplex the tokenizer withTokenizer create(java.io.InputStream toTokenize)
toTokenize
- void setTokenPreProcessor(TokenPreProcess preProcessor)
preProcessor
- the token pre processor to useTokenPreProcess getTokenPreProcessor()