org.deeplearning4j.spark.iterator

Class PathSparkDataSetIterator

java.lang.Object

org.deeplearning4j.spark.iterator.BaseDataSetIterator<java.lang.String>

org.deeplearning4j.spark.iterator.PathSparkDataSetIterator

All Implemented Interfaces:

java.io.Serializable, java.util.Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class PathSparkDataSetIterator
extends BaseDataSetIterator<java.lang.String>

A DataSetIterator that loads serialized DataSet objects (saved with DataSet.save(OutputStream)) from a String that represents the path (for example, on HDFS)

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	BUFFER_SIZE

Fields inherited from class org.deeplearning4j.spark.iterator.BaseDataSetIterator

batch, cursor, dataSetStreams, inputColumns, iter, preloadedDataSet, preprocessor, totalOutcomes

Constructor Summary

Constructors

Constructor and Description
PathSparkDataSetIterator(java.util.Collection<java.lang.String> dataSetStreams)
PathSparkDataSetIterator(java.util.Iterator<java.lang.String> iter)

Method Summary

Modifier and Type	Method and Description
protected org.nd4j.linalg.dataset.DataSet	load(java.lang.String path)
org.nd4j.linalg.dataset.DataSet	next()
int	totalExamples()

Methods inherited from class org.deeplearning4j.spark.iterator.BaseDataSetIterator

asyncSupported, batch, cursor, getLabels, getPreProcessor, hasNext, inputColumns, next, numExamples, remove, reset, resetSupported, setPreProcessor, totalOutcomes

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

BUFFER_SIZE

public static final int BUFFER_SIZE

See Also:

Constant Field Values

Constructor Detail

PathSparkDataSetIterator

public PathSparkDataSetIterator(java.util.Iterator<java.lang.String> iter)

PathSparkDataSetIterator

public PathSparkDataSetIterator(java.util.Collection<java.lang.String> dataSetStreams)

Method Detail

totalExamples

public int totalExamples()

Specified by:

totalExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Specified by:

totalExamples in class BaseDataSetIterator<java.lang.String>

next

public org.nd4j.linalg.dataset.DataSet next()

load

protected org.nd4j.linalg.dataset.DataSet load(java.lang.String path)

Specified by:

load in class BaseDataSetIterator<java.lang.String>