org.deeplearning4j.spark.models.sequencevectors.functions

Class BaseTokenizerFunction

java.lang.Object

org.deeplearning4j.spark.models.sequencevectors.functions.BaseTokenizerFunction

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

DocumentSequenceConvertFunction, KeySequenceConvertFunction, TokenizerFunction

public abstract class BaseTokenizerFunction
extends java.lang.Object
implements java.io.Serializable

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.spark.broadcast.Broadcast<VectorsConfiguration>	configurationBroadcast
protected TokenizerFactory	tokenizerFactory
protected TokenPreProcess	tokenPreprocessor

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseTokenizerFunction(org.apache.spark.broadcast.Broadcast<VectorsConfiguration> configurationBroadcast)

Method Summary

Modifier and Type	Method and Description
protected void	instantiateTokenizerFactory()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

configurationBroadcast

protected org.apache.spark.broadcast.Broadcast<VectorsConfiguration> configurationBroadcast

tokenizerFactory

protected transient TokenizerFactory tokenizerFactory

tokenPreprocessor

protected transient TokenPreProcess tokenPreprocessor

Constructor Detail

BaseTokenizerFunction

protected BaseTokenizerFunction(@NonNull
                                org.apache.spark.broadcast.Broadcast<VectorsConfiguration> configurationBroadcast)

Method Detail

instantiateTokenizerFactory

protected void instantiateTokenizerFactory()