org.deeplearning4j.text.tokenization.tokenizer.preprocessor

Class StemmingPreprocessor

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizer.preprocessor.CommonPreprocessor

org.deeplearning4j.text.tokenization.tokenizer.preprocessor.StemmingPreprocessor

All Implemented Interfaces:

TokenPreProcess

public class StemmingPreprocessor
extends CommonPreprocessor

This tokenizer preprocessor implements basic cleaning inherited from CommonPreprocessor + does english Porter stemming on tokens PLEASE NOTE: This preprocessor is thread-safe by using synchronized method

Constructor Summary

Constructors

Constructor and Description
StemmingPreprocessor()

Method Summary

Modifier and Type	Method and Description
java.lang.String	preProcess(java.lang.String token) Pre process a token

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StemmingPreprocessor

public StemmingPreprocessor()

Method Detail

preProcess

public java.lang.String preProcess(java.lang.String token)

Description copied from interface: TokenPreProcess

Pre process a token

Specified by:

preProcess in interface TokenPreProcess

Overrides:

preProcess in class CommonPreprocessor

Parameters:

token - the token to pre process

Returns:

the preprocessed token