org.deeplearning4j.text.tokenization.tokenizer

Class PosUimaTokenizer

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizer.PosUimaTokenizer

All Implemented Interfaces:

Tokenizer

public class PosUimaTokenizer
extends java.lang.Object
implements Tokenizer

Filter by part of speech tag. Any not valid part of speech tags become NONE

Constructor Summary

Constructors

Constructor and Description
PosUimaTokenizer(java.lang.String tokens, org.apache.uima.analysis_engine.AnalysisEngine engine, java.util.Collection<java.lang.String> allowedPosTags)
PosUimaTokenizer(java.lang.String tokens, org.apache.uima.analysis_engine.AnalysisEngine engine, java.util.Collection<java.lang.String> allowedPosTags, boolean stripNones)

Method Summary

Modifier and Type	Method and Description
int	countTokens() The number of tokens in the tokenizer
static org.apache.uima.analysis_engine.AnalysisEngine	defaultAnalysisEngine()
java.util.List<java.lang.String>	getTokens() Returns a list of all the tokens
boolean	hasMoreTokens() An iterator for tracking whether more tokens are left in the iterator not
java.lang.String	nextToken() The next token (word usually) in the string
void	setTokenPreProcessor(TokenPreProcess tokenPreProcessor) Set the token pre process

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PosUimaTokenizer

public PosUimaTokenizer(java.lang.String tokens,
                        org.apache.uima.analysis_engine.AnalysisEngine engine,
                        java.util.Collection<java.lang.String> allowedPosTags)

PosUimaTokenizer

public PosUimaTokenizer(java.lang.String tokens,
                        org.apache.uima.analysis_engine.AnalysisEngine engine,
                        java.util.Collection<java.lang.String> allowedPosTags,
                        boolean stripNones)

Method Detail

hasMoreTokens

public boolean hasMoreTokens()

Description copied from interface: Tokenizer

An iterator for tracking whether more tokens are left in the iterator not

Specified by:

hasMoreTokens in interface Tokenizer

Returns:

whether there is anymore tokens to iterate over

countTokens

public int countTokens()

Description copied from interface: Tokenizer

The number of tokens in the tokenizer

Specified by:

countTokens in interface Tokenizer

Returns:

the number of tokens

nextToken

public java.lang.String nextToken()

Description copied from interface: Tokenizer

The next token (word usually) in the string

Specified by:

nextToken in interface Tokenizer

Returns:

the next token in the string if any

getTokens

public java.util.List<java.lang.String> getTokens()

Description copied from interface: Tokenizer

Returns a list of all the tokens

Specified by:

getTokens in interface Tokenizer

Returns:

a list of all the tokens

defaultAnalysisEngine

public static org.apache.uima.analysis_engine.AnalysisEngine defaultAnalysisEngine()

setTokenPreProcessor

public void setTokenPreProcessor(@NonNull
                                 TokenPreProcess tokenPreProcessor)

Description copied from interface: Tokenizer

Set the token pre process

Specified by:

setTokenPreProcessor in interface Tokenizer

Parameters:

tokenPreProcessor - the token pre processor to set