Package org.datavec.nlp.tokenization.tokenizer

Interface Summary

Interface	Description
Tokenizer	A representation of a tokenizer.
TokenPreProcess	Token preprocessing

Class Summary

Class	Description
ConcurrentTokenizer	OpenNLP Tokenizer annotator.
DefaultStreamTokenizer	Tokenizer based on the StreamTokenizer
DefaultTokenizer	Default tokenizer
PosUimaTokenizer	Filter by part of speech tag.
UimaTokenizer	Tokenizer based on the passed in analysis engine