org.deeplearning4j.bagofwords.vectorizer

Class BaseTextVectorizer

java.lang.Object

org.deeplearning4j.bagofwords.vectorizer.BaseTextVectorizer

All Implemented Interfaces:

java.io.Serializable, TextVectorizer, Vectorizer

Direct Known Subclasses:

BagOfWordsVectorizer, TfidfVectorizer

public abstract class BaseTextVectorizer
extends java.lang.Object
implements TextVectorizer

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected InvertedIndex<VocabWord>	index
protected boolean	isParallel
protected LabelAwareIterator	iterator
protected LabelsSource	labelsSource
protected int	minWordFrequency
protected java.util.Collection<java.lang.String>	stopWords
protected TokenizerFactory	tokenizerFactory
protected VocabCache<VocabWord>	vocabCache

Constructor Summary

Constructors

Constructor and Description
BaseTextVectorizer()

Method Summary

Modifier and Type	Method and Description
void	buildVocab()
void	fit() Train the model
protected LabelsSource	getLabelsSource()
long	numWordsEncountered() Returns the number of words encountered so far

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.deeplearning4j.bagofwords.vectorizer.TextVectorizer

getIndex, getVocabCache, transform, transform, vectorize, vectorize, vectorize

Methods inherited from interface org.deeplearning4j.datasets.vectorizer.Vectorizer

vectorize

Field Detail

tokenizerFactory

protected transient TokenizerFactory tokenizerFactory

iterator

protected transient LabelAwareIterator iterator

minWordFrequency

protected int minWordFrequency

vocabCache

protected VocabCache<VocabWord> vocabCache

labelsSource

protected LabelsSource labelsSource

stopWords

protected java.util.Collection<java.lang.String> stopWords

index

protected transient InvertedIndex<VocabWord> index

isParallel

protected boolean isParallel

Constructor Detail

BaseTextVectorizer

public BaseTextVectorizer()

Method Detail

getLabelsSource

protected LabelsSource getLabelsSource()

buildVocab

public void buildVocab()

fit

public void fit()

Description copied from interface: TextVectorizer

Train the model

Specified by:

fit in interface TextVectorizer

numWordsEncountered

public long numWordsEncountered()

Returns the number of words encountered so far

Specified by:

numWordsEncountered in interface TextVectorizer

Returns:

the number of words encountered so far