org.datavec.nlp.metadata

Class DefaultVocabCache

java.lang.Object

org.datavec.nlp.metadata.DefaultVocabCache

All Implemented Interfaces:

VocabCache

public class DefaultVocabCache
extends java.lang.Object
implements VocabCache

Vocab cache uswed for storing information about vocab

Constructor Summary

Constructors

Constructor and Description
DefaultVocabCache()
DefaultVocabCache(int minWordFrequency) Instantiate with a given min word frequency

Method Summary

Modifier and Type	Method and Description
int	getMinWordFrequency()
double	idf(java.lang.String word) Number of documents word has occurred in
void	incrementCount(java.lang.String word) Increment a word count by 1
void	incrementCount(java.lang.String word, double by) Increment count for a word
void	incrementDocCount(java.lang.String word) Increment the doc count for a word by 1
void	incrementDocCount(java.lang.String word, double by) Increment the document count for a particular word
void	incrementNumDocs(double by) Increment the number of documents
void	initialize(Configuration conf) Configuration for initializing
int	minWordFrequency() The min word frequency needed to be included in the vocab (default 5)
double	numDocs() Number of documents
void	setMinWordFrequency(int minWordFrequency)
double	tfidf(java.lang.String word, double frequency) Calculate the tfidf of the word given the document frequency
Index	vocabWords() All of the vocab words (ordered) note that these are not all the possible tokens
java.lang.String	wordAt(int i) Returns a word in the vocab at a particular index
double	wordFrequency(java.lang.String word) Get the word frequency for a word

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DefaultVocabCache

public DefaultVocabCache(int minWordFrequency)

Instantiate with a given min word frequency

Parameters:

minWordFrequency -

DefaultVocabCache

public DefaultVocabCache()

Method Detail

incrementNumDocs

public void incrementNumDocs(double by)

Description copied from interface: VocabCache

Increment the number of documents

Specified by:

incrementNumDocs in interface VocabCache

numDocs

public double numDocs()

Description copied from interface: VocabCache

Number of documents

Specified by:

numDocs in interface VocabCache

Returns:

the number of documents

wordAt

public java.lang.String wordAt(int i)

Description copied from interface: VocabCache

Returns a word in the vocab at a particular index

Specified by:

wordAt in interface VocabCache

Parameters:

i - the index to get

Returns:

the word at that index in the vocab

initialize

public void initialize(Configuration conf)

Description copied from interface: VocabCache

Configuration for initializing

Specified by:

initialize in interface VocabCache

Parameters:

conf - the configuration to initialize with

wordFrequency

public double wordFrequency(java.lang.String word)

Description copied from interface: VocabCache

Get the word frequency for a word

Specified by:

wordFrequency in interface VocabCache

Parameters:

word - the word to get frequency for

Returns:

the frequency for a given word

minWordFrequency

public int minWordFrequency()

Description copied from interface: VocabCache

The min word frequency needed to be included in the vocab (default 5)

Specified by:

minWordFrequency in interface VocabCache

Returns:

the min word frequency to be included in the vocab

vocabWords

public Index vocabWords()

Description copied from interface: VocabCache

All of the vocab words (ordered) note that these are not all the possible tokens

Specified by:

vocabWords in interface VocabCache

Returns:

the list of vocab words

incrementDocCount

public void incrementDocCount(java.lang.String word)

Description copied from interface: VocabCache

Increment the doc count for a word by 1

Specified by:

incrementDocCount in interface VocabCache

Parameters:

word - the word to increment the count for

incrementDocCount

public void incrementDocCount(java.lang.String word,
                              double by)

Description copied from interface: VocabCache

Increment the document count for a particular word

Specified by:

incrementDocCount in interface VocabCache

Parameters:

word - the word to increment the count for

by - the amount to increment by

incrementCount

public void incrementCount(java.lang.String word)

Description copied from interface: VocabCache

Increment a word count by 1

Specified by:

incrementCount in interface VocabCache

Parameters:

word - the word to increment the count for

incrementCount

public void incrementCount(java.lang.String word,
                           double by)

Description copied from interface: VocabCache

Increment count for a word

Specified by:

incrementCount in interface VocabCache

Parameters:

word - the word to increment the count for

by - the amount to increment by

idf

public double idf(java.lang.String word)

Description copied from interface: VocabCache

Number of documents word has occurred in

Specified by:

idf in interface VocabCache

Parameters:

word - the word to get the idf for

tfidf

public double tfidf(java.lang.String word,
                    double frequency)

Description copied from interface: VocabCache

Calculate the tfidf of the word given the document frequency

Specified by:

tfidf in interface VocabCache

Parameters:

word - the word to get frequency for

frequency - the frequency

Returns:

the tfidf for a word

getMinWordFrequency

public int getMinWordFrequency()

setMinWordFrequency

public void setMinWordFrequency(int minWordFrequency)