org.deeplearning4j.models.word2vec.wordstore

Interface VocabCache<T extends SequenceElement>

All Superinterfaces:

java.io.Serializable

All Known Implementing Classes:

AbstractCache, InMemoryLookupCache

public interface VocabCache<T extends SequenceElement>
extends java.io.Serializable

A VocabCache handles the storage of information needed for the word2vec look up table.

Method Summary

Modifier and Type	Method and Description
void	addToken(T element) Adds a token to the cache
void	addWordToIndex(int index, long elementId)
void	addWordToIndex(int index, java.lang.String word)
boolean	containsWord(java.lang.String word) Returns true if the cache contains the given word
int	docAppearedIn(java.lang.String word) Count of documents a word appeared in
T	elementAtIndex(int index) Returns SequenceElement at the given index or null
boolean	hasToken(java.lang.String token) Returns whether the cache contains this token or not
void	importVocabulary(VocabCache<T> vocabCache) imports vocabulary
void	incrementDocCount(java.lang.String word, long howMuch) Increment the document count
void	incrementTotalDocCount() Increment the doc count
void	incrementTotalDocCount(long by) Increment the doc count
void	incrementWordCount(java.lang.String word) Increment the count for the given word
void	incrementWordCount(java.lang.String word, int increment) Increment the count for the given word by the amount increment
int	indexOf(java.lang.String word) Returns the index of a given word
void	loadVocab() Load vocab
int	numWords() Returns the number of words in the cache
void	putVocabWord(java.lang.String word) Deprecated.
void	removeElement(java.lang.String label) Removes element with specified label from vocabulary Please note: Huffman index should be updated after element removal
void	removeElement(T element) Removes specified element from vocabulary Please note: Huffman index should be updated after element removal
void	saveVocab() Saves the vocab: this allow for reuse of word frequencies
void	setCountForDoc(java.lang.String word, long count) Set the count for the number of documents the word appears in
T	tokenFor(long id)
T	tokenFor(java.lang.String word) Returns the token (again not necessarily in the vocab) for this word
java.util.Collection<T>	tokens() All of the tokens in the cache, (not necessarily apart of the vocab)
long	totalNumberOfDocs() Returns the total of number of documents encountered in the corpus
long	totalWordOccurrences() The total number of word occurrences
void	updateWordsOccurencies() Updates counters
boolean	vocabExists() Vocab exists already
java.util.Collection<T>	vocabWords() Returns all of the vocab word nodes
java.lang.String	wordAtIndex(int index) Returns the word contained at the given index or null
T	wordFor(long id)
T	wordFor(java.lang.String word)
int	wordFrequency(java.lang.String word) Returns the number of times the word has occurred
java.util.Collection<java.lang.String>	words() Returns all of the words in the vocab

Method Detail

loadVocab

void loadVocab()

Load vocab

vocabExists

boolean vocabExists()

Vocab exists already

Returns:

saveVocab

void saveVocab()

Saves the vocab: this allow for reuse of word frequencies

words

java.util.Collection<java.lang.String> words()

Returns all of the words in the vocab

incrementWordCount

void incrementWordCount(java.lang.String word)

Increment the count for the given word

Parameters:

word - the word to increment the count for

incrementWordCount

void incrementWordCount(java.lang.String word,
                        int increment)

Increment the count for the given word by the amount increment

Parameters:

word - the word to increment the count for

increment - the amount to increment by

wordFrequency

int wordFrequency(java.lang.String word)

Returns the number of times the word has occurred

Parameters:

word - the word to retrieve the occurrence frequency for

Returns:

0 if hasn't occurred or the number of times the word occurs

containsWord

boolean containsWord(java.lang.String word)

Returns true if the cache contains the given word

Parameters:

word - the word to check for

Returns:

wordAtIndex

java.lang.String wordAtIndex(int index)

Returns the word contained at the given index or null

Parameters:

index - the index of the word to get

Returns:

the word at the given index

elementAtIndex

T elementAtIndex(int index)

Returns SequenceElement at the given index or null

Parameters:

index -

Returns:

indexOf

int indexOf(java.lang.String word)

Returns the index of a given word

Parameters:

word - the index of a given word

Returns:

the index of a given word or -1 if not found

vocabWords

java.util.Collection<T> vocabWords()

Returns all of the vocab word nodes

Returns:

totalWordOccurrences

long totalWordOccurrences()

The total number of word occurrences

Returns:

the total number of word occurrences

wordFor

T wordFor(java.lang.String word)

Parameters:

word -

Returns:

wordFor

T wordFor(long id)

addWordToIndex

void addWordToIndex(int index,
                    java.lang.String word)

Parameters:

index -

word -

addWordToIndex

void addWordToIndex(int index,
                    long elementId)

putVocabWord

@Deprecated
void putVocabWord(java.lang.String word)

Deprecated.

Inserts the word as a vocab word (it gets the vocab word from the internal token store). Note that the index must be set on the token.

Parameters:

word - the word to add to the vocab

numWords

int numWords()

Returns the number of words in the cache

Returns:

the number of words in the cache

docAppearedIn

int docAppearedIn(java.lang.String word)

Count of documents a word appeared in

Parameters:

word - the number of documents the word appeared in

Returns:

incrementDocCount

void incrementDocCount(java.lang.String word,
                       long howMuch)

Increment the document count

Parameters:

word - the word to increment by

howMuch -

setCountForDoc

void setCountForDoc(java.lang.String word,
                    long count)

Set the count for the number of documents the word appears in

Parameters:

word - the word to set the count for

count - the count of the word

totalNumberOfDocs

long totalNumberOfDocs()

Returns the total of number of documents encountered in the corpus

Returns:

the total number of docs in the corpus

incrementTotalDocCount

void incrementTotalDocCount()

Increment the doc count

incrementTotalDocCount

void incrementTotalDocCount(long by)

Increment the doc count

Parameters:

by - the number to increment by

tokens

java.util.Collection<T> tokens()

All of the tokens in the cache, (not necessarily apart of the vocab)

Returns:

the tokens for this cache

addToken

void addToken(T element)

Adds a token to the cache

Parameters:

element - the word to add

tokenFor

T tokenFor(java.lang.String word)

Returns the token (again not necessarily in the vocab) for this word

Parameters:

word - the word to get the token for

Returns:

the vocab word for this token

tokenFor

T tokenFor(long id)

hasToken

boolean hasToken(java.lang.String token)

Returns whether the cache contains this token or not

Parameters:

token - the token to tes

Returns:

whether the token exists in the cache or not

importVocabulary

void importVocabulary(VocabCache<T> vocabCache)

imports vocabulary

Parameters:

vocabCache -

updateWordsOccurencies

void updateWordsOccurencies()

Updates counters

removeElement

void removeElement(java.lang.String label)

Removes element with specified label from vocabulary Please note: Huffman index should be updated after element removal

Parameters:

label - label of the element to be removed

removeElement

void removeElement(T element)

Removes specified element from vocabulary Please note: Huffman index should be updated after element removal

Parameters:

element - SequenceElement to be removed