org.deeplearning4j.spark.models.sequencevectors.functions

Class TokenizerFunction

java.lang.Object

org.deeplearning4j.spark.models.sequencevectors.functions.BaseTokenizerFunction

org.deeplearning4j.spark.models.sequencevectors.functions.TokenizerFunction

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.Function<java.lang.String,Sequence<VocabWord>>

public class TokenizerFunction
extends BaseTokenizerFunction
implements org.apache.spark.api.java.function.Function<java.lang.String,Sequence<VocabWord>>

See Also:

Serialized Form

Field Summary

Fields inherited from class org.deeplearning4j.spark.models.sequencevectors.functions.BaseTokenizerFunction

configurationBroadcast, tokenizerFactory, tokenPreprocessor

Constructor Summary

Constructors

Constructor and Description
TokenizerFunction(org.apache.spark.broadcast.Broadcast<VectorsConfiguration> configurationBroadcast)

Method Summary

Modifier and Type	Method and Description
Sequence<VocabWord>	call(java.lang.String s)

Methods inherited from class org.deeplearning4j.spark.models.sequencevectors.functions.BaseTokenizerFunction

instantiateTokenizerFactory

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerFunction

public TokenizerFunction(@NonNull
                         org.apache.spark.broadcast.Broadcast<VectorsConfiguration> configurationBroadcast)

Method Detail

call

public Sequence<VocabWord> call(java.lang.String s)
                         throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.Function<java.lang.String,Sequence<VocabWord>>

Throws:

java.lang.Exception