public class JapaneseTokenizer extends java.lang.Object implements Tokenizer
Constructor and Description |
---|
JapaneseTokenizer(java.lang.String toTokenize) |
Modifier and Type | Method and Description |
---|---|
int |
countTokens()
The number of tokens in the tokenizer
|
java.util.List<java.lang.String> |
getTokens()
Returns a list of all the tokens
|
boolean |
hasMoreTokens()
An iterator for tracking whether
more tokens are left in the iterator not
|
java.lang.String |
nextToken()
The next token (word usually) in the string
|
void |
setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
Set the token pre process
|
public boolean hasMoreTokens()
Tokenizer
hasMoreTokens
in interface Tokenizer
public int countTokens()
Tokenizer
countTokens
in interface Tokenizer
public java.lang.String nextToken()
Tokenizer
public java.util.List<java.lang.String> getTokens()
Tokenizer
public void setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
Tokenizer
setTokenPreProcessor
in interface Tokenizer
tokenPreProcessor
- the token pre processor to set