org.datavec.spark.functions.pairdata

Class MapToBytesPairWritableFunction

java.lang.Object

org.datavec.spark.functions.pairdata.MapToBytesPairWritableFunction

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,java.lang.Iterable<scala.Tuple3<java.lang.String,java.lang.Integer,org.apache.spark.input.PortableDataStream>>>,org.apache.hadoop.io.Text,BytesPairWritable>

public class MapToBytesPairWritableFunction
extends java.lang.Object
implements org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,java.lang.Iterable<scala.Tuple3<java.lang.String,java.lang.Integer,org.apache.spark.input.PortableDataStream>>>,org.apache.hadoop.io.Text,BytesPairWritable>

A function to read files (assuming exactly 2 per input) from a PortableDataStream and combine the contents into a BytesPairWritable

See Also:

DataVecSparkUtil.combineFilesForSequenceFile(JavaSparkContext, String, String, PathToKeyConverter, PathToKeyConverter), Serialized Form

Constructor Summary

Constructors

Constructor and Description
MapToBytesPairWritableFunction()

Method Summary

Modifier and Type	Method and Description
scala.Tuple2<org.apache.hadoop.io.Text,BytesPairWritable>	call(scala.Tuple2<java.lang.String,java.lang.Iterable<scala.Tuple3<java.lang.String,java.lang.Integer,org.apache.spark.input.PortableDataStream>>> in)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MapToBytesPairWritableFunction

public MapToBytesPairWritableFunction()

Method Detail

call

public scala.Tuple2<org.apache.hadoop.io.Text,BytesPairWritable> call(scala.Tuple2<java.lang.String,java.lang.Iterable<scala.Tuple3<java.lang.String,java.lang.Integer,org.apache.spark.input.PortableDataStream>>> in)
                                                               throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,java.lang.Iterable<scala.Tuple3<java.lang.String,java.lang.Integer,org.apache.spark.input.PortableDataStream>>>,org.apache.hadoop.io.Text,BytesPairWritable>

Throws:

java.lang.Exception