Constructor and Description |
---|
KoreanTokenizer(java.lang.String toTokenize) |
Modifier and Type | Method and Description |
---|---|
int |
countTokens()
The number of tokens in the tokenizer
|
java.util.List<java.lang.String> |
getTokens()
Returns a list of all the tokens
|
boolean |
hasMoreTokens()
An iterator for tracking whether
more tokens are left in the iterator not
|
java.lang.String |
nextToken()
The next token (word usually) in the string
|
void |
setTokenPreProcessor(TokenPreProcess tokenPreProcess)
Set the token pre process
|
public boolean hasMoreTokens()
Tokenizer
hasMoreTokens
in interface Tokenizer
public int countTokens()
Tokenizer
countTokens
in interface Tokenizer
public java.lang.String nextToken()
Tokenizer
public java.util.List<java.lang.String> getTokens()
Tokenizer
public void setTokenPreProcessor(TokenPreProcess tokenPreProcess)
Tokenizer
setTokenPreProcessor
in interface Tokenizer
tokenPreProcess
- the token pre processor to set