Constructor and Description |
---|
NGramTokenizer(Tokenizer tokenizer,
java.lang.Integer minN,
java.lang.Integer maxN) |
Modifier and Type | Method and Description |
---|---|
int |
countTokens()
The number of tokens in the tokenizer
|
java.util.List<java.lang.String> |
getTokens()
Returns a list of all the tokens
|
boolean |
hasMoreTokens()
An iterator for tracking whether
more tokens are left in the iterator not
|
java.lang.String |
nextToken()
The next token (word usually) in the string
|
void |
setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
Set the token pre process
|
public NGramTokenizer(Tokenizer tokenizer, java.lang.Integer minN, java.lang.Integer maxN)
public boolean hasMoreTokens()
Tokenizer
hasMoreTokens
in interface Tokenizer
public int countTokens()
Tokenizer
countTokens
in interface Tokenizer
public java.lang.String nextToken()
Tokenizer
public java.util.List<java.lang.String> getTokens()
Tokenizer
public void setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
Tokenizer
setTokenPreProcessor
in interface Tokenizer
tokenPreProcessor
- the token pre processor to set