org.datavec.nlp.tokenization.tokenizerfactory

Interface TokenizerFactory

All Known Implementing Classes:

DefaultTokenizerFactory, PosUimaTokenizerFactory, UimaTokenizerFactory

public interface TokenizerFactory

Generates a tokenizer for a given string

Method Summary

Modifier and Type	Method and Description
Tokenizer	create(java.io.InputStream toTokenize) Create a tokenizer based on an input stream
Tokenizer	create(java.lang.String toTokenize) The tokenizer to createComplex
void	setTokenPreProcessor(TokenPreProcess preProcessor) Sets a token pre processor to be used with every tokenizer

Method Detail

create

Tokenizer create(java.lang.String toTokenize)

The tokenizer to createComplex

Parameters:

toTokenize - the string to createComplex the tokenizer with

Returns:

the new tokenizer

create

Tokenizer create(java.io.InputStream toTokenize)

Create a tokenizer based on an input stream

Parameters:

toTokenize -

Returns:

setTokenPreProcessor

void setTokenPreProcessor(TokenPreProcess preProcessor)

Sets a token pre processor to be used with every tokenizer

Parameters:

preProcessor - the token pre processor to use