org.deeplearning4j.models.word2vec.wordstore.inmemory

Class AbstractCache<T extends SequenceElement>

java.lang.Object

org.deeplearning4j.models.word2vec.wordstore.inmemory.AbstractCache<T>

All Implemented Interfaces:

java.io.Serializable, VocabCache<T>

public class AbstractCache<T extends SequenceElement>
extends java.lang.Object
implements VocabCache<T>

This is generic VocabCache implementation designed as abstract SequenceElements vocabulary

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AbstractCache.Builder<T extends SequenceElement>

Constructor Summary

Constructors

Constructor and Description
AbstractCache()

Method Summary

Modifier and Type	Method and Description
void	addToken(T element) This method adds specified SequenceElement to vocabulary
void	addWordToIndex(int index, long elementId)
void	addWordToIndex(int index, java.lang.String label) This method allows to insert specified label to specified Huffman tree position.
boolean	containsElement(T element) Checks, if specified element exists in vocabulary
boolean	containsWord(java.lang.String word) Checks, if specified label exists in vocabulary
int	docAppearedIn(java.lang.String word) Returns number of documents (if applicable) the label was observed in.
T	elementAtIndex(int index) Returns SequenceElement at specified index
boolean	hasToken(java.lang.String label) Checks, if specified label already exists in vocabulary.
void	importVocabulary(VocabCache<T> vocabCache) This method imports all elements from VocabCache passed as argument If element already exists,
void	incrementDocCount(java.lang.String word, long howMuch) Increment number of documents the label was observed in Please note: this method is NOT thread-safe
void	incrementTotalDocCount() Increment total number of documents observed by 1
void	incrementTotalDocCount(long by) Increment total number of documents observed by specified value
void	incrementWordCount(java.lang.String word) Increment frequency for specified label by 1
void	incrementWordCount(java.lang.String word, int increment) Increment frequency for specified label by specified value
int	indexOf(java.lang.String label) Returns Huffman index for specified label
void	loadVocab() Deserialize vocabulary from specified path
int	numWords() Returns number of elements in this vocabulary
void	putVocabWord(java.lang.String word) Deprecated.
void	removeElement(java.lang.String label) Removes element with specified label from vocabulary Please note: Huffman index should be updated after element removal
void	removeElement(T element) Removes specified element from vocabulary Please note: Huffman index should be updated after element removal
void	saveVocab() Serialize vocabulary to specified path
void	setCountForDoc(java.lang.String word, long count) Set exact number of observed documents that contain specified word Please note: this method is NOT thread-safe
T	tokenFor(long id)
T	tokenFor(java.lang.String label) Returns SequenceElement for specified label.
java.util.Collection<T>	tokens() Returns collection of SequenceElements from this vocabulary.
long	totalNumberOfDocs() Returns total number of documents observed (if applicable)
long	totalWordOccurrences() Returns total number of elements observed
void	updateWordsOccurencies() Updates counters
boolean	vocabExists() Returns true, if number of elements in vocabulary > 0, false otherwise
java.util.Collection<T>	vocabWords() Returns collection of SequenceElements stored in this vocabulary
java.lang.String	wordAtIndex(int index) Returns the label of the element at specified Huffman index
T	wordFor(long id)
T	wordFor(java.lang.String label) Returns SequenceElement for specified label
int	wordFrequency(java.lang.String word) Returns the SequenceElement's frequency over training corpus
java.util.Collection<java.lang.String>	words() Returns collection of labels available in this vocabulary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbstractCache

public AbstractCache()

Method Detail

loadVocab

public void loadVocab()

Deserialize vocabulary from specified path

Specified by:

loadVocab in interface VocabCache<T extends SequenceElement>

vocabExists

public boolean vocabExists()

Returns true, if number of elements in vocabulary > 0, false otherwise

Specified by:

vocabExists in interface VocabCache<T extends SequenceElement>

Returns:

saveVocab

public void saveVocab()

Serialize vocabulary to specified path

Specified by:

saveVocab in interface VocabCache<T extends SequenceElement>

words

public java.util.Collection<java.lang.String> words()

Returns collection of labels available in this vocabulary

Specified by:

words in interface VocabCache<T extends SequenceElement>

Returns:

incrementWordCount

public void incrementWordCount(java.lang.String word)

Increment frequency for specified label by 1

Specified by:

incrementWordCount in interface VocabCache<T extends SequenceElement>

Parameters:

word - the word to increment the count for

incrementWordCount

public void incrementWordCount(java.lang.String word,
                               int increment)

Increment frequency for specified label by specified value

Specified by:

incrementWordCount in interface VocabCache<T extends SequenceElement>

Parameters:

word - the word to increment the count for

increment - the amount to increment by

wordFrequency

public int wordFrequency(@NonNull
                         java.lang.String word)

Returns the SequenceElement's frequency over training corpus

Specified by:

wordFrequency in interface VocabCache<T extends SequenceElement>

Parameters:

word - the word to retrieve the occurrence frequency for

Returns:

containsWord

public boolean containsWord(java.lang.String word)

Checks, if specified label exists in vocabulary

Specified by:

containsWord in interface VocabCache<T extends SequenceElement>

Parameters:

word - the word to check for

Returns:

containsElement

public boolean containsElement(T element)

Checks, if specified element exists in vocabulary

Parameters:

element -

Returns:

wordAtIndex

public java.lang.String wordAtIndex(int index)

Returns the label of the element at specified Huffman index

Specified by:

wordAtIndex in interface VocabCache<T extends SequenceElement>

Parameters:

index - the index of the word to get

Returns:

elementAtIndex

public T elementAtIndex(int index)

Returns SequenceElement at specified index

Specified by:

elementAtIndex in interface VocabCache<T extends SequenceElement>

Parameters:

index -

Returns:

indexOf

public int indexOf(java.lang.String label)

Returns Huffman index for specified label

Specified by:

indexOf in interface VocabCache<T extends SequenceElement>

Parameters:

label - the label to get index for

Returns:

>=0 if label exists, -1 if Huffman tree wasn't built yet, -2 if specified label wasn't found

vocabWords

public java.util.Collection<T> vocabWords()

Returns collection of SequenceElements stored in this vocabulary

Specified by:

vocabWords in interface VocabCache<T extends SequenceElement>

Returns:

totalWordOccurrences

public long totalWordOccurrences()

Returns total number of elements observed

Specified by:

totalWordOccurrences in interface VocabCache<T extends SequenceElement>

Returns:

wordFor

public T wordFor(@NonNull
                 java.lang.String label)

Returns SequenceElement for specified label

Specified by:

wordFor in interface VocabCache<T extends SequenceElement>

Parameters:

label - to fetch element for

Returns:

wordFor

public T wordFor(long id)

Specified by:

wordFor in interface VocabCache<T extends SequenceElement>

addWordToIndex

public void addWordToIndex(int index,
                           java.lang.String label)

This method allows to insert specified label to specified Huffman tree position. CAUTION: Never use this, unless you 100% sure what are you doing.

Specified by:

addWordToIndex in interface VocabCache<T extends SequenceElement>

Parameters:

index -

label -

addWordToIndex

public void addWordToIndex(int index,
                           long elementId)

Specified by:

addWordToIndex in interface VocabCache<T extends SequenceElement>

putVocabWord

@Deprecated
public void putVocabWord(java.lang.String word)

Deprecated.

Description copied from interface: VocabCache

Inserts the word as a vocab word (it gets the vocab word from the internal token store). Note that the index must be set on the token.

Specified by:

putVocabWord in interface VocabCache<T extends SequenceElement>

Parameters:

word - the word to add to the vocab

numWords

public int numWords()

Returns number of elements in this vocabulary

Specified by:

numWords in interface VocabCache<T extends SequenceElement>

Returns:

docAppearedIn

public int docAppearedIn(java.lang.String word)

Returns number of documents (if applicable) the label was observed in.

Specified by:

docAppearedIn in interface VocabCache<T extends SequenceElement>

Parameters:

word - the number of documents the word appeared in

Returns:

incrementDocCount

public void incrementDocCount(java.lang.String word,
                              long howMuch)

Increment number of documents the label was observed in Please note: this method is NOT thread-safe

Specified by:

incrementDocCount in interface VocabCache<T extends SequenceElement>

Parameters:

word - the word to increment by

howMuch -

setCountForDoc

public void setCountForDoc(java.lang.String word,
                           long count)

Set exact number of observed documents that contain specified word Please note: this method is NOT thread-safe

Specified by:

setCountForDoc in interface VocabCache<T extends SequenceElement>

Parameters:

word - the word to set the count for

count - the count of the word

totalNumberOfDocs

public long totalNumberOfDocs()

Returns total number of documents observed (if applicable)

Specified by:

totalNumberOfDocs in interface VocabCache<T extends SequenceElement>

Returns:

incrementTotalDocCount

public void incrementTotalDocCount()

Increment total number of documents observed by 1

Specified by:

incrementTotalDocCount in interface VocabCache<T extends SequenceElement>

incrementTotalDocCount

public void incrementTotalDocCount(long by)

Increment total number of documents observed by specified value

Specified by:

incrementTotalDocCount in interface VocabCache<T extends SequenceElement>

Parameters:

by - the number to increment by

tokens

public java.util.Collection<T> tokens()

Returns collection of SequenceElements from this vocabulary. The same as vocabWords() method

Specified by:

tokens in interface VocabCache<T extends SequenceElement>

Returns:

collection of SequenceElements

addToken

public void addToken(T element)

This method adds specified SequenceElement to vocabulary

Specified by:

addToken in interface VocabCache<T extends SequenceElement>

Parameters:

element - the word to add

tokenFor

public T tokenFor(java.lang.String label)

Returns SequenceElement for specified label. The same as wordFor() method.

Specified by:

tokenFor in interface VocabCache<T extends SequenceElement>

Parameters:

label - the label to get the token for

Returns:

tokenFor

public T tokenFor(long id)

Specified by:

tokenFor in interface VocabCache<T extends SequenceElement>

hasToken

public boolean hasToken(java.lang.String label)

Checks, if specified label already exists in vocabulary. The same as containsWord() method.

Specified by:

hasToken in interface VocabCache<T extends SequenceElement>

Parameters:

label - the token to test

Returns:

importVocabulary

public void importVocabulary(@NonNull
                             VocabCache<T> vocabCache)

This method imports all elements from VocabCache passed as argument If element already exists,

Specified by:

importVocabulary in interface VocabCache<T extends SequenceElement>

Parameters:

vocabCache -

updateWordsOccurencies

public void updateWordsOccurencies()

Description copied from interface: VocabCache

Updates counters

Specified by:

updateWordsOccurencies in interface VocabCache<T extends SequenceElement>

removeElement

public void removeElement(java.lang.String label)

Description copied from interface: VocabCache

Removes element with specified label from vocabulary Please note: Huffman index should be updated after element removal

Specified by:

removeElement in interface VocabCache<T extends SequenceElement>

Parameters:

label - label of the element to be removed

removeElement

public void removeElement(T element)

Description copied from interface: VocabCache

Removes specified element from vocabulary Please note: Huffman index should be updated after element removal

Specified by:

removeElement in interface VocabCache<T extends SequenceElement>

Parameters:

element - SequenceElement to be removed