org.deeplearning4j.spark.data

Class BatchAndExportMultiDataSetsFunction

java.lang.Object

org.deeplearning4j.spark.data.BatchAndExportMultiDataSetsFunction

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>,java.util.Iterator<java.lang.String>>

public class BatchAndExportMultiDataSetsFunction
extends java.lang.Object
implements org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>,java.util.Iterator<java.lang.String>>

Function used with RDD<MultiDataSet>.mapPartitionsWithIndex. It does two things: 1. Batch MultiDataSets together, to the specified minibatch size. This may result in splitting or combining existing MultiDataSet objects as required 2. Export the MultiDataSet objects to the specified directory.

Naming convention for exported files: "mds_" + partitionIdx + JVM_UID + "_" + idx + ".bin" where 'idx' is the index of the MultiDataSet objects in this partition

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
BatchAndExportMultiDataSetsFunction(int minibatchSize, java.lang.String exportBaseDirectory)

Method Summary

Modifier and Type	Method and Description
java.util.Iterator<java.lang.String>	call(java.lang.Integer partitionIdx, java.util.Iterator<org.nd4j.linalg.dataset.api.MultiDataSet> iterator)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BatchAndExportMultiDataSetsFunction

public BatchAndExportMultiDataSetsFunction(int minibatchSize,
                                           java.lang.String exportBaseDirectory)

Parameters:

minibatchSize - Minibatch size to combine examples to (if necessary)

exportBaseDirectory - Base directory for exporting

Method Detail

call

public java.util.Iterator<java.lang.String> call(java.lang.Integer partitionIdx,
                                                 java.util.Iterator<org.nd4j.linalg.dataset.api.MultiDataSet> iterator)
                                          throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>,java.util.Iterator<java.lang.String>>

Throws:

java.lang.Exception