org.deeplearning4j.spark.data

Class BatchAndExportDataSetsFunction

java.lang.Object

org.deeplearning4j.spark.data.BatchAndExportDataSetsFunction

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<org.nd4j.linalg.dataset.DataSet>,java.util.Iterator<java.lang.String>>

public class BatchAndExportDataSetsFunction
extends java.lang.Object
implements org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<org.nd4j.linalg.dataset.DataSet>,java.util.Iterator<java.lang.String>>

Function used with RDD<DataSet>.mapPartitionsWithIndex. It does two things: 1. Batch DataSets together, to the specified minibatch size. This may result in splitting or combining existing DataSet objects as required 2. Export the DataSet objects to the specified directory.

Naming convention for exported files: "dataset_" + partitionIdx + JVM_UID + "_" + idx + ".bin" where 'idx' is the index of the DataSet objects in this partition

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
BatchAndExportDataSetsFunction(int minibatchSize, java.lang.String exportBaseDirectory)

Method Summary

Modifier and Type	Method and Description
java.util.Iterator<java.lang.String>	call(java.lang.Integer partitionIdx, java.util.Iterator<org.nd4j.linalg.dataset.DataSet> iterator)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BatchAndExportDataSetsFunction

public BatchAndExportDataSetsFunction(int minibatchSize,
                                      java.lang.String exportBaseDirectory)

Parameters:

minibatchSize - Minibatch size to combine examples to (if necessary)

exportBaseDirectory - Base directory for exporting

Method Detail

call

public java.util.Iterator<java.lang.String> call(java.lang.Integer partitionIdx,
                                                 java.util.Iterator<org.nd4j.linalg.dataset.DataSet> iterator)
                                          throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<org.nd4j.linalg.dataset.DataSet>,java.util.Iterator<java.lang.String>>

Throws:

java.lang.Exception