org.deeplearning4j.spark.iterator

Class PortableDataStreamDataSetIterator

java.lang.Object

org.deeplearning4j.spark.iterator.BaseDataSetIterator<org.apache.spark.input.PortableDataStream>

org.deeplearning4j.spark.iterator.PortableDataStreamDataSetIterator

All Implemented Interfaces:

java.io.Serializable, java.util.Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class PortableDataStreamDataSetIterator
extends BaseDataSetIterator<org.apache.spark.input.PortableDataStream>

A DataSetIterator that loads serialized DataSet objects (saved with DataSet.save(OutputStream)) from a PortableDataStream, usually obtained from SparkContext.binaryFiles()

See Also:

Serialized Form

Field Summary

Fields inherited from class org.deeplearning4j.spark.iterator.BaseDataSetIterator

batch, cursor, dataSetStreams, inputColumns, iter, preloadedDataSet, preprocessor, totalOutcomes

Constructor Summary

Constructors

Constructor and Description
PortableDataStreamDataSetIterator(java.util.Collection<org.apache.spark.input.PortableDataStream> dataSetStreams)
PortableDataStreamDataSetIterator(java.util.Iterator<org.apache.spark.input.PortableDataStream> iter)

Method Summary

Modifier and Type	Method and Description
protected org.nd4j.linalg.dataset.DataSet	load(org.apache.spark.input.PortableDataStream pds)
org.nd4j.linalg.dataset.DataSet	next()
int	totalExamples()

Methods inherited from class org.deeplearning4j.spark.iterator.BaseDataSetIterator

asyncSupported, batch, cursor, getLabels, getPreProcessor, hasNext, inputColumns, next, numExamples, remove, reset, resetSupported, setPreProcessor, totalOutcomes

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Constructor Detail

PortableDataStreamDataSetIterator

public PortableDataStreamDataSetIterator(java.util.Iterator<org.apache.spark.input.PortableDataStream> iter)

PortableDataStreamDataSetIterator

public PortableDataStreamDataSetIterator(java.util.Collection<org.apache.spark.input.PortableDataStream> dataSetStreams)

Method Detail

totalExamples

public int totalExamples()

Specified by:

totalExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Specified by:

totalExamples in class BaseDataSetIterator<org.apache.spark.input.PortableDataStream>

next

public org.nd4j.linalg.dataset.DataSet next()

load

protected org.nd4j.linalg.dataset.DataSet load(org.apache.spark.input.PortableDataStream pds)

Specified by:

load in class BaseDataSetIterator<org.apache.spark.input.PortableDataStream>