org.deeplearning4j.text.tokenization.tokenizerfactory

Class UimaTokenizerFactory

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizerfactory.UimaTokenizerFactory

All Implemented Interfaces:

TokenizerFactory

public class UimaTokenizerFactory
extends java.lang.Object
implements TokenizerFactory

Uses a uima AnalysisEngine to tokenize text.

Constructor Summary

Constructors

Constructor and Description
UimaTokenizerFactory()
UimaTokenizerFactory(org.apache.uima.analysis_engine.AnalysisEngine tokenizer)
UimaTokenizerFactory(org.apache.uima.analysis_engine.AnalysisEngine tokenizer, boolean checkForLabel)
UimaTokenizerFactory(boolean checkForLabel)
UimaTokenizerFactory(UimaResource resource)
UimaTokenizerFactory(UimaResource resource, boolean checkForLabel)

Method Summary

Modifier and Type	Method and Description
Tokenizer	create(java.io.InputStream toTokenize) Create a tokenizer based on an input stream
Tokenizer	create(java.lang.String toTokenize) The tokenizer to createComplex
static org.apache.uima.analysis_engine.AnalysisEngine	defaultAnalysisEngine() Creates a tokenization,/stemming pipeline
TokenPreProcess	getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance
UimaResource	getUimaResource()
void	setTokenPreProcessor(TokenPreProcess preProcessor) Sets a token pre processor to be used with every tokenizer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

UimaTokenizerFactory

public UimaTokenizerFactory()
                     throws org.apache.uima.resource.ResourceInitializationException

Throws:

org.apache.uima.resource.ResourceInitializationException

UimaTokenizerFactory

public UimaTokenizerFactory(UimaResource resource)

UimaTokenizerFactory

public UimaTokenizerFactory(org.apache.uima.analysis_engine.AnalysisEngine tokenizer)

UimaTokenizerFactory

public UimaTokenizerFactory(UimaResource resource,
                            boolean checkForLabel)

UimaTokenizerFactory

public UimaTokenizerFactory(boolean checkForLabel)
                     throws org.apache.uima.resource.ResourceInitializationException

Throws:

org.apache.uima.resource.ResourceInitializationException

UimaTokenizerFactory

public UimaTokenizerFactory(org.apache.uima.analysis_engine.AnalysisEngine tokenizer,
                            boolean checkForLabel)

Method Detail

create

public Tokenizer create(java.lang.String toTokenize)

Description copied from interface: TokenizerFactory

The tokenizer to createComplex

Specified by:

create in interface TokenizerFactory

Parameters:

toTokenize - the string to createComplex the tokenizer with

Returns:

the new tokenizer

getUimaResource

public UimaResource getUimaResource()

defaultAnalysisEngine

public static org.apache.uima.analysis_engine.AnalysisEngine defaultAnalysisEngine()

Creates a tokenization,/stemming pipeline

Returns:

a tokenization/stemming pipeline

create

public Tokenizer create(java.io.InputStream toTokenize)

Description copied from interface: TokenizerFactory

Create a tokenizer based on an input stream

Specified by:

create in interface TokenizerFactory

Returns:

setTokenPreProcessor

public void setTokenPreProcessor(TokenPreProcess preProcessor)

Description copied from interface: TokenizerFactory

Sets a token pre processor to be used with every tokenizer

Specified by:

setTokenPreProcessor in interface TokenizerFactory

Parameters:

preProcessor - the token pre processor to use

getTokenPreProcessor

public TokenPreProcess getTokenPreProcessor()

Returns TokenPreProcessor set for this TokenizerFactory instance

Specified by:

getTokenPreProcessor in interface TokenizerFactory

Returns:

TokenPreProcessor instance, or null if no preprocessor was defined