org.datavec.spark.functions.pairdata

Class PathToKeyFunction

java.lang.Object

org.datavec.spark.functions.pairdata.PathToKeyFunction

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,org.apache.spark.input.PortableDataStream>,java.lang.String,scala.Tuple3<java.lang.String,java.lang.Integer,org.apache.spark.input.PortableDataStream>>

public class PathToKeyFunction
extends java.lang.Object
implements org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,org.apache.spark.input.PortableDataStream>,java.lang.String,scala.Tuple3<java.lang.String,java.lang.Integer,org.apache.spark.input.PortableDataStream>>

Given a Tuple2, where the first value is the full path, map this to a Tuple3 where the first value is a key (using a PathToKeyConverter), second is an index, and third is the original data stream

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
PathToKeyFunction(int index, PathToKeyConverter converter)

Method Summary

Modifier and Type	Method and Description
scala.Tuple2<java.lang.String,scala.Tuple3<java.lang.String,java.lang.Integer,org.apache.spark.input.PortableDataStream>>	call(scala.Tuple2<java.lang.String,org.apache.spark.input.PortableDataStream> in)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PathToKeyFunction

public PathToKeyFunction(int index,
                         PathToKeyConverter converter)

Method Detail

call

public scala.Tuple2<java.lang.String,scala.Tuple3<java.lang.String,java.lang.Integer,org.apache.spark.input.PortableDataStream>> call(scala.Tuple2<java.lang.String,org.apache.spark.input.PortableDataStream> in)
                                                                                                                               throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,org.apache.spark.input.PortableDataStream>,java.lang.String,scala.Tuple3<java.lang.String,java.lang.Integer,org.apache.spark.input.PortableDataStream>>

Throws:

java.lang.Exception