org.deeplearning4j.models.word2vec.wordstore

Class VocabConstructor<T extends SequenceElement>

java.lang.Object

org.deeplearning4j.models.word2vec.wordstore.VocabConstructor<T>

public class VocabConstructor<T extends SequenceElement>
extends java.lang.Object

This class can be used to build joint vocabulary from special sources, that should be treated separately. I.e. words from one source should have minWordFrequency set to 1, while the rest of corpus should have minWordFrequency set to 5. So, here's the way to deal with it. It also can be used to simply build vocabulary out of arbitrary number of Sequences derived from arbitrary number of SequenceIterators

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	VocabConstructor.Builder<T extends SequenceElement>
protected class	VocabConstructor.VocabRunnable

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	log

Method Summary

Modifier and Type	Method and Description
protected WeightLookupTable<T>	buildExtendedLookupTable() Placeholder for future implementation
protected VocabCache<T>	buildExtendedVocabulary() Placeholder for future implementation
VocabCache<T>	buildJointVocabulary(boolean resetCounters, boolean buildHuffmanTree) This method scans all sources passed through builder, and returns all words as vocab.
VocabCache<T>	buildMergedVocabulary(VocabCache<T> vocabCache, boolean fetchLabels) This method transfers existing vocabulary into current one Please note: this method expects source vocabulary has Huffman tree indexes applied
VocabCache<T>	buildMergedVocabulary(WordVectors wordVectors, boolean fetchLabels) This method transfers existing WordVectors model into current one
protected void	filterVocab(AbstractCache<T> cache, int minWordFrequency)
long	getNumberOfSequences() This method returns total number of sequences passed through VocabConstructor

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.slf4j.Logger log

Method Detail

buildExtendedLookupTable

protected WeightLookupTable<T> buildExtendedLookupTable()

Placeholder for future implementation

Returns:

buildExtendedVocabulary

protected VocabCache<T> buildExtendedVocabulary()

Placeholder for future implementation

Returns:

buildMergedVocabulary

public VocabCache<T> buildMergedVocabulary(@NonNull
                                           WordVectors wordVectors,
                                           boolean fetchLabels)

This method transfers existing WordVectors model into current one

Parameters:

wordVectors -

Returns:

getNumberOfSequences

public long getNumberOfSequences()

This method returns total number of sequences passed through VocabConstructor

Returns:

buildMergedVocabulary

public VocabCache<T> buildMergedVocabulary(@NonNull
                                           VocabCache<T> vocabCache,
                                           boolean fetchLabels)

This method transfers existing vocabulary into current one Please note: this method expects source vocabulary has Huffman tree indexes applied

Parameters:

vocabCache -

Returns:

buildJointVocabulary

public VocabCache<T> buildJointVocabulary(boolean resetCounters,
                                          boolean buildHuffmanTree)

This method scans all sources passed through builder, and returns all words as vocab. If TargetVocabCache was set during instance creation, it'll be filled too.

Returns:

filterVocab

protected void filterVocab(AbstractCache<T> cache,
                           int minWordFrequency)