org.deeplearning4j.datasets.iterator

Class AsyncDataSetIterator

java.lang.Object

org.deeplearning4j.datasets.iterator.AsyncDataSetIterator

All Implemented Interfaces:

java.io.Serializable, java.util.Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class AsyncDataSetIterator
extends java.lang.Object
implements org.nd4j.linalg.dataset.api.iterator.DataSetIterator

AsyncDataSetIterator takes an existing DataSetIterator and loads one or more DataSet objects from it using a separate thread. For data sets where DataSetIterator.next() is long running (limited by disk read or processing time for example) this may improve performance by loading the next DataSet asynchronously (i.e., while training is continuing on the previous DataSet). Obviously this may use additional memory.
Note however that due to asynchronous loading of data, next(int) is not supported.

PLEASE NOTE: If used together with CUDA backend, please use it with caution.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	logger

Constructor Summary

Constructors

Constructor and Description
AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator) Create an AsyncDataSetIterator with a queue size of 1 (i.e., only load a single additional DataSet)
AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator, int queueSize) Create an AsyncDataSetIterator with a specified queue size.
AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator, int queueSize, java.util.concurrent.BlockingQueue<org.nd4j.linalg.dataset.DataSet> queue) Create an AsyncDataSetIterator with a queue size of 1 (i.e., only load a single additional DataSet)

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported()
int	batch()
int	cursor()
java.util.List<java.lang.String>	getLabels()
org.nd4j.linalg.dataset.api.DataSetPreProcessor	getPreProcessor()
boolean	hasNext()
int	inputColumns()
org.nd4j.linalg.dataset.DataSet	next()
org.nd4j.linalg.dataset.DataSet	next(int num)
int	numExamples()
void	remove()
void	reset()
boolean	resetSupported()
void	setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)
void	shutdown() Shut down the async data set iterator thread This is not typically necessary if using a single AsyncDataSetIterator (thread is a daemon thread and so shouldn't block the JVM from exiting) Behaviour of next(), hasNext() etc methods after shutdown of async iterator is undefined
int	totalExamples()
int	totalOutcomes()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

logger

protected static final org.slf4j.Logger logger

Constructor Detail

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator)

Create an AsyncDataSetIterator with a queue size of 1 (i.e., only load a single additional DataSet)

Parameters:

baseIterator - The DataSetIterator to load data from asynchronously

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator,
                            int queueSize,
                            java.util.concurrent.BlockingQueue<org.nd4j.linalg.dataset.DataSet> queue)

Create an AsyncDataSetIterator with a queue size of 1 (i.e., only load a single additional DataSet)

Parameters:

iterator -

queueSize -

queue - BlockingQueue instance that will be used as backing queue. MagicQueue probably?

AsyncDataSetIterator

public AsyncDataSetIterator(org.nd4j.linalg.dataset.api.iterator.DataSetIterator baseIterator,
                            int queueSize)

Create an AsyncDataSetIterator with a specified queue size. LinkedBlockingQueue will be used as backing queue

Parameters:

baseIterator - The DataSetIterator to load data from asynchronously

queueSize - size of the queue (max number of elements to load into queue)

Method Detail

next

public org.nd4j.linalg.dataset.DataSet next(int num)

Specified by:

next in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalExamples

public int totalExamples()

Specified by:

totalExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

inputColumns

public int inputColumns()

Specified by:

inputColumns in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalOutcomes

public int totalOutcomes()

Specified by:

totalOutcomes in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

resetSupported

public boolean resetSupported()

Specified by:

resetSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

asyncSupported

public boolean asyncSupported()

Specified by:

asyncSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

reset

public void reset()

Specified by:

reset in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

batch

public int batch()

Specified by:

batch in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

cursor

public int cursor()

Specified by:

cursor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

numExamples

public int numExamples()

Specified by:

numExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

setPreProcessor

public void setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)

Specified by:

setPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

getPreProcessor

public org.nd4j.linalg.dataset.api.DataSetPreProcessor getPreProcessor()

Specified by:

getPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

getLabels

public java.util.List<java.lang.String> getLabels()

Specified by:

getLabels in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

hasNext

public boolean hasNext()

Specified by:

hasNext in interface java.util.Iterator<org.nd4j.linalg.dataset.DataSet>

next

public org.nd4j.linalg.dataset.DataSet next()

Specified by:

next in interface java.util.Iterator<org.nd4j.linalg.dataset.DataSet>

shutdown

public void shutdown()

Shut down the async data set iterator thread This is not typically necessary if using a single AsyncDataSetIterator (thread is a daemon thread and so shouldn't block the JVM from exiting) Behaviour of next(), hasNext() etc methods after shutdown of async iterator is undefined

remove

public void remove()

Specified by:

remove in interface java.util.Iterator<org.nd4j.linalg.dataset.DataSet>