org.datavec.nlp.vectorizer

Class AbstractTfidfVectorizer<VECTOR_TYPE>

java.lang.Object

org.datavec.nlp.vectorizer.TextVectorizer<VECTOR_TYPE>

org.datavec.nlp.vectorizer.AbstractTfidfVectorizer<VECTOR_TYPE>

All Implemented Interfaces:

Vectorizer<VECTOR_TYPE>

Direct Known Subclasses:

TfidfVectorizer

public abstract class AbstractTfidfVectorizer<VECTOR_TYPE>
extends TextVectorizer<VECTOR_TYPE>

Tf idf vectorizer

Nested Class Summary

Nested classes/interfaces inherited from interface org.datavec.api.vector.Vectorizer

Vectorizer.RecordCallBack

Field Summary

Fields inherited from class org.datavec.nlp.vectorizer.TextVectorizer

cache, MIN_WORD_FREQUENCY, minWordFrequency, STOP_WORDS, stopWords, TOKENIZER, tokenizerFactory, VOCAB_CACHE

Constructor Summary

Constructors

Constructor and Description
AbstractTfidfVectorizer()

Method Summary

Modifier and Type	Method and Description
TokenizerFactory	createTokenizerFactory(Configuration conf) Create tokenizer factory based on the configuration
abstract VECTOR_TYPE	createVector(java.lang.Object[] args) Create a vector based on the given arguments
void	doWithTokens(Tokenizer tokenizer) Increment counts, add to collection,...
abstract VECTOR_TYPE	fitTransform(RecordReader reader) Fit based on a record reader
abstract VECTOR_TYPE	transform(Record record) Transform a record in to a vector

Methods inherited from class org.datavec.nlp.vectorizer.TextVectorizer

fit, fit, initialize, toString, wordFrequenciesForRecord

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.datavec.api.vector.Vectorizer

fitTransform

Constructor Detail

AbstractTfidfVectorizer

public AbstractTfidfVectorizer()

Method Detail

doWithTokens

public void doWithTokens(Tokenizer tokenizer)

Description copied from class: TextVectorizer

Increment counts, add to collection,...

Specified by:

doWithTokens in class TextVectorizer<VECTOR_TYPE>

createTokenizerFactory

public TokenizerFactory createTokenizerFactory(Configuration conf)

Description copied from class: TextVectorizer

Create tokenizer factory based on the configuration

Specified by:

createTokenizerFactory in class TextVectorizer<VECTOR_TYPE>

Parameters:

conf - the configuration to use

Returns:

the tokenizer factory based on the configuration

createVector

public abstract VECTOR_TYPE createVector(java.lang.Object[] args)

Description copied from interface: Vectorizer

Create a vector based on the given arguments

Parameters:

args - the arguments to create a vector with

Returns:

the created vector

fitTransform

public abstract VECTOR_TYPE fitTransform(RecordReader reader)

Description copied from interface: Vectorizer

Fit based on a record reader

transform

public abstract VECTOR_TYPE transform(Record record)

Description copied from interface: Vectorizer

Transform a record in to a vector

Parameters:

record - the record to write

Returns: