org.deeplearning4j.text.tokenization.tokenizerfactory

Class PosUimaTokenizerFactory

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizerfactory.PosUimaTokenizerFactory

All Implemented Interfaces:

TokenizerFactory

public class PosUimaTokenizerFactory
extends java.lang.Object
implements TokenizerFactory

Creates a tokenizer that filters by part of speech tags

See Also:

{org.deeplearning4j.text.tokenization.tokenizer.PosUimaTokenizer}

Constructor Summary

Constructors

Constructor and Description
PosUimaTokenizerFactory(org.apache.uima.analysis_engine.AnalysisEngine tokenizer, java.util.Collection<java.lang.String> allowedPosTags)
PosUimaTokenizerFactory(java.util.Collection<java.lang.String> allowedPoSTags)
PosUimaTokenizerFactory(java.util.Collection<java.lang.String> allowedPoSTags, boolean stripNones)

Method Summary

Modifier and Type	Method and Description
Tokenizer	create(java.io.InputStream toTokenize) Create a tokenizer based on an input stream
Tokenizer	create(java.lang.String toTokenize) The tokenizer to createComplex
static org.apache.uima.analysis_engine.AnalysisEngine	defaultAnalysisEngine()
TokenPreProcess	getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance
void	setTokenPreProcessor(TokenPreProcess preProcessor) Sets a token pre processor to be used with every tokenizer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PosUimaTokenizerFactory

public PosUimaTokenizerFactory(java.util.Collection<java.lang.String> allowedPoSTags,
                               boolean stripNones)

PosUimaTokenizerFactory

public PosUimaTokenizerFactory(java.util.Collection<java.lang.String> allowedPoSTags)

PosUimaTokenizerFactory

public PosUimaTokenizerFactory(org.apache.uima.analysis_engine.AnalysisEngine tokenizer,
                               java.util.Collection<java.lang.String> allowedPosTags)

Method Detail

defaultAnalysisEngine

public static org.apache.uima.analysis_engine.AnalysisEngine defaultAnalysisEngine()

create

public Tokenizer create(java.lang.String toTokenize)

Description copied from interface: TokenizerFactory

The tokenizer to createComplex

Specified by:

create in interface TokenizerFactory

Parameters:

toTokenize - the string to createComplex the tokenizer with

Returns:

the new tokenizer

create

public Tokenizer create(java.io.InputStream toTokenize)

Description copied from interface: TokenizerFactory

Create a tokenizer based on an input stream

Specified by:

create in interface TokenizerFactory

Returns:

setTokenPreProcessor

public void setTokenPreProcessor(TokenPreProcess preProcessor)

Description copied from interface: TokenizerFactory

Sets a token pre processor to be used with every tokenizer

Specified by:

setTokenPreProcessor in interface TokenizerFactory

Parameters:

preProcessor - the token pre processor to use

getTokenPreProcessor

public TokenPreProcess getTokenPreProcessor()

Returns TokenPreProcessor set for this TokenizerFactory instance

Specified by:

getTokenPreProcessor in interface TokenizerFactory

Returns:

TokenPreProcessor instance, or null if no preprocessor was defined