org.datavec.nlp.vectorizer

Class TextVectorizer<VECTOR_TYPE>

java.lang.Object

org.datavec.nlp.vectorizer.TextVectorizer<VECTOR_TYPE>

All Implemented Interfaces:

Vectorizer<VECTOR_TYPE>

Direct Known Subclasses:

AbstractTfidfVectorizer

public abstract class TextVectorizer<VECTOR_TYPE>
extends java.lang.Object
implements Vectorizer<VECTOR_TYPE>

Baseline text vectorizer that includes some common elements to text analysis such as the tokenizer factory

Nested Class Summary

Nested classes/interfaces inherited from interface org.datavec.api.vector.Vectorizer

Vectorizer.RecordCallBack

Field Summary

Fields

Modifier and Type	Field and Description
protected VocabCache	cache
static java.lang.String	MIN_WORD_FREQUENCY
protected int	minWordFrequency
static java.lang.String	STOP_WORDS
protected java.util.Collection<java.lang.String>	stopWords
static java.lang.String	TOKENIZER
protected TokenizerFactory	tokenizerFactory
static java.lang.String	VOCAB_CACHE

Constructor Summary

Constructors

Constructor and Description
TextVectorizer()

Method Summary

Modifier and Type	Method and Description
abstract TokenizerFactory	createTokenizerFactory(Configuration conf) Create tokenizer factory based on the configuration
abstract void	doWithTokens(Tokenizer tokenizer) Increment counts, add to collection,...
void	fit(RecordReader reader) Fit based on a record reader
void	fit(RecordReader reader, Vectorizer.RecordCallBack callBack) Fit based on a record reader
void	initialize(Configuration conf) Initialize based on a configuration
protected java.lang.String	toString(java.util.Collection<Writable> record)
protected Counter<java.lang.String>	wordFrequenciesForRecord(java.util.Collection<Writable> record)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.datavec.api.vector.Vectorizer

createVector, fitTransform, fitTransform, transform

Field Detail

tokenizerFactory

protected TokenizerFactory tokenizerFactory

minWordFrequency

protected int minWordFrequency

MIN_WORD_FREQUENCY

public static final java.lang.String MIN_WORD_FREQUENCY

See Also:

Constant Field Values

STOP_WORDS

public static final java.lang.String STOP_WORDS

See Also:

Constant Field Values

TOKENIZER

public static final java.lang.String TOKENIZER

See Also:

Constant Field Values

VOCAB_CACHE

public static final java.lang.String VOCAB_CACHE

See Also:

Constant Field Values

stopWords

protected java.util.Collection<java.lang.String> stopWords

cache

protected VocabCache cache

Constructor Detail

TextVectorizer

public TextVectorizer()

Method Detail

initialize

public void initialize(Configuration conf)

Description copied from interface: Vectorizer

Initialize based on a configuration

Specified by:

initialize in interface Vectorizer<VECTOR_TYPE>

Parameters:

conf - the configuration to use

fit

public void fit(RecordReader reader)

Description copied from interface: Vectorizer

Fit based on a record reader

Specified by:

fit in interface Vectorizer<VECTOR_TYPE>

fit

public void fit(RecordReader reader,
                Vectorizer.RecordCallBack callBack)

Description copied from interface: Vectorizer

Fit based on a record reader

Specified by:

fit in interface Vectorizer<VECTOR_TYPE>

wordFrequenciesForRecord

protected Counter<java.lang.String> wordFrequenciesForRecord(java.util.Collection<Writable> record)

toString

protected java.lang.String toString(java.util.Collection<Writable> record)

doWithTokens

public abstract void doWithTokens(Tokenizer tokenizer)

Increment counts, add to collection,...

Parameters:

tokenizer -

createTokenizerFactory

public abstract TokenizerFactory createTokenizerFactory(Configuration conf)

Create tokenizer factory based on the configuration

Parameters:

conf - the configuration to use

Returns:

the tokenizer factory based on the configuration