org.deeplearning4j.spark.impl.common

Class SplitPartitionsFunction<T>

java.lang.Object

org.deeplearning4j.spark.impl.common.SplitPartitionsFunction<T>

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<T>,java.util.Iterator<T>>

public class SplitPartitionsFunction<T>
extends java.lang.Object
implements org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<T>,java.util.Iterator<T>>

SplitPartitionsFunction is used to split a RDD (using AbstractJavaRDDLike.mapPartitionsWithIndex(Function2, boolean) via filtering.
It is similar in design to JavaRDD.randomSplit(double[]) however it is less prone to producing imbalanced splits that that method. Specifically, JavaRDD.randomSplit(double[]) splts each element individually (i.e., randomly determine a new split for each element at random), whereas this method chooses one out of every numSplits objects per output split. Exactly which of these is done randomly.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
SplitPartitionsFunction()

Method Summary

Modifier and Type	Method and Description
java.util.Iterator<T>	call(java.lang.Integer v1, java.util.Iterator<T> iter)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SplitPartitionsFunction

public SplitPartitionsFunction()

Method Detail

call

public java.util.Iterator<T> call(java.lang.Integer v1,
                                  java.util.Iterator<T> iter)
                           throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<T>,java.util.Iterator<T>>

Throws:

java.lang.Exception