org.datavec.nlp.reader

Class TfidfRecordReader

java.lang.Object

org.datavec.api.records.reader.BaseRecordReader

org.datavec.api.records.reader.impl.FileRecordReader

org.datavec.nlp.reader.TfidfRecordReader

All Implemented Interfaces:

java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable, Configurable, RecordReader

public class TfidfRecordReader
extends FileRecordReader

TFIDF record reader (wraps a tfidf vectorizer for delivering labels and conforming to the record reader interface)

See Also:

Serialized Form

Field Summary

Fields inherited from class org.datavec.api.records.reader.impl.FileRecordReader

appendLabel, conf, currentFile, inputSplit, iter, labels

Fields inherited from class org.datavec.api.records.reader.BaseRecordReader

listeners

Fields inherited from interface org.datavec.api.records.reader.RecordReader

APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors

Constructor and Description
TfidfRecordReader()

Method Summary

Modifier and Type	Method and Description
void	close()
Configuration	getConf() Return the configuration used by this object.
int	getNumFeatures()
TfidfVectorizer	getTfidfVectorizer()
boolean	hasNext() Whether there are anymore records
void	initialize(Configuration conf, InputSplit split) Called once at initialization.
void	initialize(InputSplit split) Called once at initialization.
java.util.List<Record>	loadFromMetaData(java.util.List<RecordMetaData> recordMetaDatas) Load multiple records from the given a list of RecordMetaData instances
Record	loadFromMetaData(RecordMetaData recordMetaData) Load a single record from the given RecordMetaData instance Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using RecordReader.loadFromMetaData(List)
java.util.List<Writable>	next() Get the next record
Record	nextRecord() Similar to RecordReader.next(), but returns a Record object, that may include metadata such as the source of the data
void	reset() Reset record reader iterator
void	setConf(Configuration conf) Set the configuration to be used by this object.
void	setTfidfVectorizer(TfidfVectorizer tfidfVectorizer)
void	shuffle()
void	shuffle(java.util.Random random)

Methods inherited from class org.datavec.api.records.reader.impl.FileRecordReader

doInitialize, getCurrentLabel, getLabels, record, setLabels

Methods inherited from class org.datavec.api.records.reader.BaseRecordReader

getListeners, invokeListeners, setListeners, setListeners

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TfidfRecordReader

public TfidfRecordReader()

Method Detail

initialize

public void initialize(InputSplit split)
                throws java.io.IOException,
                       java.lang.InterruptedException

Description copied from interface: RecordReader

Called once at initialization.

Specified by:

initialize in interface RecordReader

Overrides:

initialize in class FileRecordReader

Parameters:

split - the split that defines the range of records to read

Throws:

java.io.IOException

java.lang.InterruptedException

initialize

public void initialize(Configuration conf,
                       InputSplit split)
                throws java.io.IOException,
                       java.lang.InterruptedException

Description copied from interface: RecordReader

Called once at initialization.

Specified by:

initialize in interface RecordReader

Overrides:

initialize in class FileRecordReader

Parameters:

conf - a configuration for initialization

split - the split that defines the range of records to read

Throws:

java.io.IOException

java.lang.InterruptedException

reset

public void reset()

Description copied from interface: RecordReader

Reset record reader iterator

Specified by:

reset in interface RecordReader

Overrides:

reset in class FileRecordReader

nextRecord

public Record nextRecord()

Description copied from interface: RecordReader

Similar to RecordReader.next(), but returns a Record object, that may include metadata such as the source of the data

Specified by:

nextRecord in interface RecordReader

Overrides:

nextRecord in class FileRecordReader

Returns:

next record

next

public java.util.List<Writable> next()

Description copied from interface: RecordReader

Get the next record

Specified by:

next in interface RecordReader

Overrides:

next in class FileRecordReader

Returns:

hasNext

public boolean hasNext()

Description copied from interface: RecordReader

Whether there are anymore records

Specified by:

hasNext in interface RecordReader

Overrides:

hasNext in class FileRecordReader

Returns:

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Overrides:

close in class FileRecordReader

Throws:

java.io.IOException

setConf

public void setConf(Configuration conf)

Description copied from interface: Configurable

Set the configuration to be used by this object.

Specified by:

setConf in interface Configurable

Overrides:

setConf in class FileRecordReader

getConf

public Configuration getConf()

Description copied from interface: Configurable

Return the configuration used by this object.

Specified by:

getConf in interface Configurable

Overrides:

getConf in class FileRecordReader

getTfidfVectorizer

public TfidfVectorizer getTfidfVectorizer()

setTfidfVectorizer

public void setTfidfVectorizer(TfidfVectorizer tfidfVectorizer)

getNumFeatures

public int getNumFeatures()

shuffle

public void shuffle()

shuffle

public void shuffle(java.util.Random random)

loadFromMetaData

public Record loadFromMetaData(RecordMetaData recordMetaData)
                        throws java.io.IOException

Description copied from interface: RecordReader

Load a single record from the given RecordMetaData instance
Note: that for data that isn't splittable (i.e., text data that needs to be scanned/split), it is more efficient to load multiple records at once using RecordReader.loadFromMetaData(List)

Specified by:

loadFromMetaData in interface RecordReader

Overrides:

loadFromMetaData in class FileRecordReader

Parameters:

recordMetaData - Metadata for the record that we want to load from

Returns:

Single record for the given RecordMetaData instance

Throws:

java.io.IOException - If I/O error occurs during loading

loadFromMetaData

public java.util.List<Record> loadFromMetaData(java.util.List<RecordMetaData> recordMetaDatas)
                                        throws java.io.IOException

Description copied from interface: RecordReader

Load multiple records from the given a list of RecordMetaData instances

Specified by:

loadFromMetaData in interface RecordReader

Overrides:

loadFromMetaData in class FileRecordReader

Parameters:

recordMetaDatas - Metadata for the records that we want to load from

Returns:

Multiple records for the given RecordMetaData instances

Throws:

java.io.IOException - If I/O error occurs during loading