org.deeplearning4j.spark.impl.common.repartition

Class AssignIndexFunction<T>

java.lang.Object

org.deeplearning4j.spark.impl.common.repartition.AssignIndexFunction<T>

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<T>,java.util.Iterator<scala.Tuple2<java.lang.Integer,T>>>

public class AssignIndexFunction<T>
extends java.lang.Object
implements org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<T>,java.util.Iterator<scala.Tuple2<java.lang.Integer,T>>>

A Function2 used to assign each element in a RDD an index (integer key). This is used later in the BalancedPartitioner to enable partitioning to be done in a way that is more reliable (less random) than standard .repartition calls

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
AssignIndexFunction(int[] partitionElementStartIdxs)

Method Summary

Modifier and Type	Method and Description
java.util.Iterator<scala.Tuple2<java.lang.Integer,T>>	call(java.lang.Integer partionNum, java.util.Iterator<T> v2)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AssignIndexFunction

public AssignIndexFunction(int[] partitionElementStartIdxs)

Parameters:

partitionElementStartIdxs - These are the start indexes for elements in each partition (determined from the number of elements in each partition). Thus length of the array must be equal to the number of partitions

Method Detail

call

public java.util.Iterator<scala.Tuple2<java.lang.Integer,T>> call(java.lang.Integer partionNum,
                                                                  java.util.Iterator<T> v2)
                                                           throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.Function2<java.lang.Integer,java.util.Iterator<T>,java.util.Iterator<scala.Tuple2<java.lang.Integer,T>>>

Throws:

java.lang.Exception