org.deeplearning4j.spark.datavec

Class DataVecDataSetFunction

java.lang.Object

org.deeplearning4j.spark.datavec.DataVecDataSetFunction

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.Function<java.util.List<org.datavec.api.writable.Writable>,org.nd4j.linalg.dataset.DataSet>

public class DataVecDataSetFunction
extends java.lang.Object
implements org.apache.spark.api.java.function.Function<java.util.List<org.datavec.api.writable.Writable>,org.nd4j.linalg.dataset.DataSet>, java.io.Serializable

Map Collection<Writable> objects (out of a datavec-spark record reader function) to DataSet objects for Spark training. Analogous to RecordReaderDataSetIterator, but in the context of Spark.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	batchSize

Constructor Summary

Constructors

Constructor and Description
DataVecDataSetFunction(int labelIndex, int numPossibleLabels, boolean regression)
DataVecDataSetFunction(int labelIndex, int numPossibleLabels, boolean regression, org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor, org.datavec.api.io.WritableConverter converter)
DataVecDataSetFunction(int labelIndexFrom, int labelIndexTo, int numPossibleLabels, boolean regression, org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor, org.datavec.api.io.WritableConverter converter) Main constructor, including for multi-label regression

Method Summary

Modifier and Type	Method and Description
org.nd4j.linalg.dataset.DataSet	call(java.util.List<org.datavec.api.writable.Writable> currList)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

batchSize

protected int batchSize

Constructor Detail

DataVecDataSetFunction

public DataVecDataSetFunction(int labelIndex,
                              int numPossibleLabels,
                              boolean regression)

DataVecDataSetFunction

public DataVecDataSetFunction(int labelIndex,
                              int numPossibleLabels,
                              boolean regression,
                              org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor,
                              org.datavec.api.io.WritableConverter converter)

Parameters:

labelIndex - Index of the label column

numPossibleLabels - Number of classes for classification (not used if regression = true)

regression - False for classification, true for regression

preProcessor - DataSetPreprocessor (may be null)

converter - WritableConverter (may be null)

DataVecDataSetFunction

public DataVecDataSetFunction(int labelIndexFrom,
                              int labelIndexTo,
                              int numPossibleLabels,
                              boolean regression,
                              org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor,
                              org.datavec.api.io.WritableConverter converter)

Main constructor, including for multi-label regression

Parameters:

labelIndexFrom - Index of the first target

labelIndexTo - Index of the last target, inclusive (for classification or single-output regression: same as labelIndexFrom)

numPossibleLabels - Unused for regression, or number of classes for classification

regression - If true: regression. false: classification

Method Detail

call

public org.nd4j.linalg.dataset.DataSet call(java.util.List<org.datavec.api.writable.Writable> currList)
                                     throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.Function<java.util.List<org.datavec.api.writable.Writable>,org.nd4j.linalg.dataset.DataSet>

Throws:

java.lang.Exception