org.deeplearning4j.text.tokenization.tokenizer.preprocessor

Class CustomStemmingPreprocessor

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizer.preprocessor.CommonPreprocessor

org.deeplearning4j.text.tokenization.tokenizer.preprocessor.CustomStemmingPreprocessor

All Implemented Interfaces:

TokenPreProcess

public class CustomStemmingPreprocessor
extends CommonPreprocessor

This is StemmingPreprocessor compatible with different StemmingProcessors defined as lucene/tartarus SnowballProgram Like, but not limited to: RussianStemmer, DutchStemmer, FrenchStemmer etc PLEASE NOTE: This preprocessor is thread-safe by using synchronized method

Constructor Summary

Constructors

Constructor and Description
CustomStemmingPreprocessor(org.tartarus.snowball.SnowballProgram stemmer)

Method Summary

Modifier and Type	Method and Description
java.lang.String	preProcess(java.lang.String token) Pre process a token

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CustomStemmingPreprocessor

public CustomStemmingPreprocessor(@NonNull
                                  org.tartarus.snowball.SnowballProgram stemmer)

Method Detail

preProcess

public java.lang.String preProcess(java.lang.String token)

Description copied from interface: TokenPreProcess

Pre process a token

Specified by:

preProcess in interface TokenPreProcess

Overrides:

preProcess in class CommonPreprocessor

Parameters:

token - the token to pre process

Returns:

the preprocessed token