org.deeplearning4j.text.tokenization.tokenizerfactory

Class KoreanTokenizerFactory

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizerfactory.KoreanTokenizerFactory

All Implemented Interfaces:

TokenizerFactory

public class KoreanTokenizerFactory
extends java.lang.Object
implements TokenizerFactory

Created by kepricon on 16. 10. 20.

Constructor Summary

Constructors

Constructor and Description
KoreanTokenizerFactory()

Method Summary

Modifier and Type	Method and Description
Tokenizer	create(java.io.InputStream inputStream) Create a tokenizer based on an input stream
Tokenizer	create(java.lang.String toTokenize) The tokenizer to createComplex
TokenPreProcess	getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance
void	setTokenPreProcessor(TokenPreProcess tokenPreProcess) Sets a token pre processor to be used with every tokenizer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KoreanTokenizerFactory

public KoreanTokenizerFactory()

Method Detail

create

public Tokenizer create(java.lang.String toTokenize)

Description copied from interface: TokenizerFactory

The tokenizer to createComplex

Specified by:

create in interface TokenizerFactory

Parameters:

toTokenize - the string to createComplex the tokenizer with

Returns:

the new tokenizer

create

public Tokenizer create(java.io.InputStream inputStream)

Description copied from interface: TokenizerFactory

Create a tokenizer based on an input stream

Specified by:

create in interface TokenizerFactory

Returns:

setTokenPreProcessor

public void setTokenPreProcessor(TokenPreProcess tokenPreProcess)

Description copied from interface: TokenizerFactory

Sets a token pre processor to be used with every tokenizer

Specified by:

setTokenPreProcessor in interface TokenizerFactory

Parameters:

tokenPreProcess - the token pre processor to use

getTokenPreProcessor

public TokenPreProcess getTokenPreProcessor()

Description copied from interface: TokenizerFactory

Returns TokenPreProcessor set for this TokenizerFactory instance

Specified by:

getTokenPreProcessor in interface TokenizerFactory

Returns:

TokenPreProcessor instance, or null if no preprocessor was defined