org.deeplearning4j.text.tokenization.tokenizer.preprocessor

Class EmbeddedStemmingPreprocessor

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizer.preprocessor.EmbeddedStemmingPreprocessor

All Implemented Interfaces:

TokenPreProcess

public class EmbeddedStemmingPreprocessor
extends java.lang.Object
implements TokenPreProcess

This tokenizer preprocessor uses given preprocessor + does english Porter stemming on tokens on top of it

Constructor Summary

Constructors

Constructor and Description
EmbeddedStemmingPreprocessor(TokenPreProcess preProcess)

Method Summary

Modifier and Type	Method and Description
java.lang.String	preProcess(java.lang.String token) Pre process a token

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EmbeddedStemmingPreprocessor

public EmbeddedStemmingPreprocessor(@NonNull
                                    TokenPreProcess preProcess)

Method Detail

preProcess

public java.lang.String preProcess(java.lang.String token)

Description copied from interface: TokenPreProcess

Pre process a token

Specified by:

preProcess in interface TokenPreProcess

Parameters:

token - the token to pre process

Returns:

the preprocessed token