org.deeplearning4j.spark.datavec

Class DataVecByteDataSetFunction

java.lang.Object

org.deeplearning4j.spark.datavec.DataVecByteDataSetFunction

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>,java.lang.Double,org.nd4j.linalg.dataset.DataSet>

public class DataVecByteDataSetFunction
extends java.lang.Object
implements org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>,java.lang.Double,org.nd4j.linalg.dataset.DataSet>

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DataVecByteDataSetFunction(int labelIndex, int numPossibleLabels, int batchSize, int byteFileLen)
DataVecByteDataSetFunction(int labelIndex, int numPossibleLabels, int batchSize, int byteFileLen, boolean regression)
DataVecByteDataSetFunction(int labelIndex, int numPossibleLabels, int batchSize, int byteFileLen, boolean regression, org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)

Method Summary

Modifier and Type	Method and Description
scala.Tuple2<java.lang.Double,org.nd4j.linalg.dataset.DataSet>	call(scala.Tuple2<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable> inputTuple)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataVecByteDataSetFunction

public DataVecByteDataSetFunction(int labelIndex,
                                  int numPossibleLabels,
                                  int batchSize,
                                  int byteFileLen)

DataVecByteDataSetFunction

public DataVecByteDataSetFunction(int labelIndex,
                                  int numPossibleLabels,
                                  int batchSize,
                                  int byteFileLen,
                                  boolean regression)

DataVecByteDataSetFunction

public DataVecByteDataSetFunction(int labelIndex,
                                  int numPossibleLabels,
                                  int batchSize,
                                  int byteFileLen,
                                  boolean regression,
                                  org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)

Parameters:

labelIndex - Index of the label column

numPossibleLabels - Number of classes for classification (not used if regression = true)

batchSize - size of examples in DataSet. Pass in total examples if including all

byteFileLen - number of bytes per individual file

regression - False for classification, true for regression

preProcessor - DataSetPreprocessor (may be null)

Method Detail

call

public scala.Tuple2<java.lang.Double,org.nd4j.linalg.dataset.DataSet> call(scala.Tuple2<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable> inputTuple)
                                                                    throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>,java.lang.Double,org.nd4j.linalg.dataset.DataSet>

Throws:

java.lang.Exception