org.datavec.nlp.tokenization.tokenizer

Class ConcurrentTokenizer

java.lang.Object

org.apache.uima.analysis_component.AnalysisComponent_ImplBase

org.apache.uima.analysis_component.Annotator_ImplBase

org.apache.uima.analysis_component.CasAnnotator_ImplBase

opennlp.uima.tokenize.AbstractTokenizer

org.datavec.nlp.tokenization.tokenizer.ConcurrentTokenizer

All Implemented Interfaces:

org.apache.uima.analysis_component.AnalysisComponent

public class ConcurrentTokenizer
extends opennlp.uima.tokenize.AbstractTokenizer

OpenNLP Tokenizer annotator.

Mandatory parameters

Type	Name	Description
String	opennlp.uima.ModelName	The name of the model file
String	opennlp.uima.SentenceType	The full name of the sentence type
String	opennlp.uima.TokenType	The full name of the token type

Optional parameters

Type	Name	Description
String	opennlp.uima.ProbabilityFeature	The name of the double probability feature (not applyTransformToDestination by default)

See Also:

TokenizerME}

Field Summary

Fields inherited from class opennlp.uima.tokenize.AbstractTokenizer

context, logger, name, sentenceType, tokenType

Constructor Summary

Constructors

Constructor and Description
ConcurrentTokenizer() Initializes a new instance.

Method Summary

Modifier and Type	Method and Description
void	destroy() Releases allocated resources.
void	initialize(org.apache.uima.UimaContext context) Initializes the current instance with the given context.
protected void	postProcessAnnotations(opennlp.tools.util.Span[] tokens, org.apache.uima.cas.text.AnnotationFS[] tokenAnnotations)
void	process(org.apache.uima.cas.CAS cas)
protected opennlp.tools.util.Span[]	tokenize(org.apache.uima.cas.CAS cas, org.apache.uima.cas.text.AnnotationFS sentence)
void	typeSystemInit(org.apache.uima.cas.TypeSystem typeSystem) Initializes the type system.

Methods inherited from class org.apache.uima.analysis_component.CasAnnotator_ImplBase

getRequiredCasInterface, process

Methods inherited from class org.apache.uima.analysis_component.Annotator_ImplBase

getCasInstancesRequired, hasNext, next

Methods inherited from class org.apache.uima.analysis_component.AnalysisComponent_ImplBase

batchProcessComplete, collectionProcessComplete, getContext, getResultSpecification, reconfigure, setResultSpecification

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ConcurrentTokenizer

public ConcurrentTokenizer()

Initializes a new instance. Note: Use initialize(UimaContext) to initialize this instance. Not use the constructor.

Method Detail

process

public void process(org.apache.uima.cas.CAS cas)
             throws org.apache.uima.analysis_engine.AnalysisEngineProcessException

Overrides:

process in class opennlp.uima.tokenize.AbstractTokenizer

Throws:

org.apache.uima.analysis_engine.AnalysisEngineProcessException

initialize

public void initialize(org.apache.uima.UimaContext context)
                throws org.apache.uima.resource.ResourceInitializationException

Initializes the current instance with the given context. Note: Do all initialization in this method, do not use the constructor.

Specified by:

initialize in interface org.apache.uima.analysis_component.AnalysisComponent

Overrides:

initialize in class opennlp.uima.tokenize.AbstractTokenizer

Throws:

org.apache.uima.resource.ResourceInitializationException

typeSystemInit

public void typeSystemInit(org.apache.uima.cas.TypeSystem typeSystem)
                    throws org.apache.uima.analysis_engine.AnalysisEngineProcessException

Initializes the type system.

Overrides:

typeSystemInit in class opennlp.uima.tokenize.AbstractTokenizer

Throws:

org.apache.uima.analysis_engine.AnalysisEngineProcessException

tokenize

protected opennlp.tools.util.Span[] tokenize(org.apache.uima.cas.CAS cas,
                                             org.apache.uima.cas.text.AnnotationFS sentence)

Specified by:

tokenize in class opennlp.uima.tokenize.AbstractTokenizer

postProcessAnnotations

protected void postProcessAnnotations(opennlp.tools.util.Span[] tokens,
                                      org.apache.uima.cas.text.AnnotationFS[] tokenAnnotations)

Overrides:

postProcessAnnotations in class opennlp.uima.tokenize.AbstractTokenizer

destroy

public void destroy()

Releases allocated resources.

Specified by:

destroy in interface org.apache.uima.analysis_component.AnalysisComponent

Overrides:

destroy in class org.apache.uima.analysis_component.AnalysisComponent_ImplBase