org.deeplearning4j.text.invertedindex

Interface InvertedIndex<T extends SequenceElement>

All Superinterfaces:

java.io.Serializable

public interface InvertedIndex<T extends SequenceElement>
extends java.io.Serializable

An inverted index for mapping words to documents and documents to words

Method Summary

Modifier and Type	Method and Description
void	addLabelForDoc(int doc, java.lang.String label) Adds words to the given document
void	addLabelForDoc(int doc, T word) Add word to a document
void	addLabelsForDoc(int doc, java.util.Collection<java.lang.String> label) Adds words to the given document
void	addLabelsForDoc(int doc, java.util.List<T> word) Add word to a document
void	addWordsToDoc(int doc, java.util.List<T> words) Adds words to the given document
void	addWordsToDoc(int doc, java.util.List<T> words, java.util.Collection<java.lang.String> label) Adds words to the given document
void	addWordsToDoc(int doc, java.util.List<T> words, java.lang.String label) Adds words to the given document
void	addWordsToDoc(int doc, java.util.List<T> words, T label) Adds words to the given document
void	addWordsToDocVocabWord(int doc, java.util.List<T> words, java.util.Collection<T> label) Adds words to the given document
void	addWordToDoc(int doc, T word) Add word to a document
int[]	allDocs() Returns a list of all documents
java.util.Iterator<java.util.List<java.util.List<T>>>	batchIter(int batchSize) Iterate over batches
int	batchSize() For word vectors, this is the batch size for which to train on
void	cleanup() Cleanup any resources used
java.util.Iterator<java.util.List<T>>	docs() Iterate over documents
java.util.List<T>	document(int index) Returns a list of words for a document
int[]	documents(T vocabWord) Returns the list of documents a vocab word is in
Pair<java.util.List<T>,java.lang.String>	documentWithLabel(int index) Returns a list of words for a document and the associated label
Pair<java.util.List<T>,java.util.Collection<java.lang.String>>	documentWithLabels(int index) Returns a list of words associated with the document and the associated labels
void	eachDoc(com.google.common.base.Function<java.util.List<T>,java.lang.Void> func, java.util.concurrent.ExecutorService exec) Iterate over each document
void	eachDocWithLabel(com.google.common.base.Function<Pair<java.util.List<T>,java.lang.String>,java.lang.Void> func, java.util.concurrent.ExecutorService exec) Iterate over each document with a label
void	eachDocWithLabels(com.google.common.base.Function<Pair<java.util.List<T>,java.util.Collection<java.lang.String>>,java.lang.Void> func, java.util.concurrent.ExecutorService exec) Iterate over each document with a label
void	finish() Finishes saving data
java.util.Iterator<java.util.List<T>>	miniBatches() Iterates over mini batches
int	numDocuments() Returns the number of documents
double	sample() Sampling for creating mini batches
long	totalWords() Total number of words in the index
void	unlock() Unlock the index

Method Detail

batchIter

java.util.Iterator<java.util.List<java.util.List<T>>> batchIter(int batchSize)

Iterate over batches

Returns:

the batch size

docs

java.util.Iterator<java.util.List<T>> docs()

Iterate over documents

Returns:

unlock

void unlock()

Unlock the index

cleanup

void cleanup()

Cleanup any resources used

sample

double sample()

Sampling for creating mini batches

Returns:

the sampling for mini batches

miniBatches

java.util.Iterator<java.util.List<T>> miniBatches()

Iterates over mini batches

Returns:

the mini batches created by this vectorizer

document

java.util.List<T> document(int index)

Returns a list of words for a document

Parameters:

index -

Returns:

documentWithLabel

Pair<java.util.List<T>,java.lang.String> documentWithLabel(int index)

Returns a list of words for a document and the associated label

Parameters:

index -

Returns:

documentWithLabels

Pair<java.util.List<T>,java.util.Collection<java.lang.String>> documentWithLabels(int index)

Returns a list of words associated with the document and the associated labels

Parameters:

index -

Returns:

documents

int[] documents(T vocabWord)

Returns the list of documents a vocab word is in

Parameters:

vocabWord - the vocab word to get documents for

Returns:

the documents for a vocab word

numDocuments

int numDocuments()

Returns the number of documents

Returns:

allDocs

int[] allDocs()

Returns a list of all documents

Returns:

the list of all documents

addWordToDoc

void addWordToDoc(int doc,
                  T word)

Add word to a document

Parameters:

doc - the document to add to

word - the word to add

addWordsToDoc

void addWordsToDoc(int doc,
                   java.util.List<T> words)

Adds words to the given document

Parameters:

doc - the document to add to

words - the words to add

addLabelForDoc

void addLabelForDoc(int doc,
                    T word)

Add word to a document

Parameters:

doc - the document to add to

word - the word to add

addLabelForDoc

void addLabelForDoc(int doc,
                    java.lang.String label)

Adds words to the given document

Parameters:

doc - the document to add to

addWordsToDoc

void addWordsToDoc(int doc,
                   java.util.List<T> words,
                   java.lang.String label)

Adds words to the given document

Parameters:

doc - the document to add to

words - the words to add

label - the label for the document

addWordsToDoc

void addWordsToDoc(int doc,
                   java.util.List<T> words,
                   T label)

Adds words to the given document

Parameters:

doc - the document to add to

words - the words to add

label - the label for the document

addLabelsForDoc

void addLabelsForDoc(int doc,
                     java.util.List<T> word)

Add word to a document

Parameters:

doc - the document to add to

word - the word to add

addLabelsForDoc

void addLabelsForDoc(int doc,
                     java.util.Collection<java.lang.String> label)

Adds words to the given document

Parameters:

doc - the document to add to

label - the labels to add

addWordsToDoc

void addWordsToDoc(int doc,
                   java.util.List<T> words,
                   java.util.Collection<java.lang.String> label)

Adds words to the given document

Parameters:

doc - the document to add to

words - the words to add

label - the label for the document

addWordsToDocVocabWord

void addWordsToDocVocabWord(int doc,
                            java.util.List<T> words,
                            java.util.Collection<T> label)

Adds words to the given document

Parameters:

doc - the document to add to

words - the words to add

label - the label for the document

finish

void finish()

Finishes saving data

totalWords

long totalWords()

Total number of words in the index

Returns:

the total number of words in the index

batchSize

int batchSize()

For word vectors, this is the batch size for which to train on

Returns:

the batch size for which to train on

eachDocWithLabels

void eachDocWithLabels(com.google.common.base.Function<Pair<java.util.List<T>,java.util.Collection<java.lang.String>>,java.lang.Void> func,
                       java.util.concurrent.ExecutorService exec)

Iterate over each document with a label

Parameters:

func - the function to apply

exec - executor service for execution

eachDocWithLabel

void eachDocWithLabel(com.google.common.base.Function<Pair<java.util.List<T>,java.lang.String>,java.lang.Void> func,
                      java.util.concurrent.ExecutorService exec)

Iterate over each document with a label

Parameters:

func - the function to apply

exec - executor service for execution

eachDoc

void eachDoc(com.google.common.base.Function<java.util.List<T>,java.lang.Void> func,
             java.util.concurrent.ExecutorService exec)

Iterate over each document

Parameters:

func - the function to apply

exec - executor service for execution