org.deeplearning4j.spark.datavec

Class DataVecSequencePairDataSetFunction

java.lang.Object

org.deeplearning4j.spark.datavec.DataVecSequencePairDataSetFunction

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.api.java.function.Function<scala.Tuple2<java.util.List<java.util.List<org.datavec.api.writable.Writable>>,java.util.List<java.util.List<org.datavec.api.writable.Writable>>>,org.nd4j.linalg.dataset.DataSet>

public class DataVecSequencePairDataSetFunction
extends java.lang.Object
implements org.apache.spark.api.java.function.Function<scala.Tuple2<java.util.List<java.util.List<org.datavec.api.writable.Writable>>,java.util.List<java.util.List<org.datavec.api.writable.Writable>>>,org.nd4j.linalg.dataset.DataSet>, java.io.Serializable

Map Tuple2<Collection<Collection<Writable>>,Collection<Collection<Writable>> objects (out of a TWO datavec-spark sequence record reader functions) to DataSet objects for Spark training. Analogous to SequenceRecordReaderDataSetIterator, but in the context of Spark. Supports loading data from a TWO sources only; hence supports many-to-one and one-to-many situations. see DataVecSequenceDataSetFunction for the single file version

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataVecSequencePairDataSetFunction.AlignmentMode Alignment mode for dealing with input/labels of differing lengths (for example, one-to-many and many-to-one type situations).

Constructor Summary

Constructors

Constructor and Description
DataVecSequencePairDataSetFunction() Constructor for equal length and no conversion of labels (i.e., regression or already in one-hot representation).
DataVecSequencePairDataSetFunction(int numPossibleLabels, boolean regression) Constructor for equal length, no data set preprocessor or writable converter
DataVecSequencePairDataSetFunction(int numPossibleLabels, boolean regression, DataVecSequencePairDataSetFunction.AlignmentMode alignmentMode) Constructor for data with a specified alignment mode, no data set preprocessor or writable converter
DataVecSequencePairDataSetFunction(int numPossibleLabels, boolean regression, DataVecSequencePairDataSetFunction.AlignmentMode alignmentMode, org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor, org.datavec.api.io.WritableConverter converter)

Method Summary

Modifier and Type	Method and Description
org.nd4j.linalg.dataset.DataSet	call(scala.Tuple2<java.util.List<java.util.List<org.datavec.api.writable.Writable>>,java.util.List<java.util.List<org.datavec.api.writable.Writable>>> input)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataVecSequencePairDataSetFunction

public DataVecSequencePairDataSetFunction()

Constructor for equal length and no conversion of labels (i.e., regression or already in one-hot representation). No data set proprocessor or writable converter

DataVecSequencePairDataSetFunction

public DataVecSequencePairDataSetFunction(int numPossibleLabels,
                                          boolean regression)

Constructor for equal length, no data set preprocessor or writable converter

See Also:

DataVecSequencePairDataSetFunction(int, boolean, AlignmentMode, DataSetPreProcessor, WritableConverter)

DataVecSequencePairDataSetFunction

public DataVecSequencePairDataSetFunction(int numPossibleLabels,
                                          boolean regression,
                                          DataVecSequencePairDataSetFunction.AlignmentMode alignmentMode)

Constructor for data with a specified alignment mode, no data set preprocessor or writable converter

See Also:

DataVecSequencePairDataSetFunction(int, boolean, AlignmentMode, DataSetPreProcessor, WritableConverter)

DataVecSequencePairDataSetFunction

public DataVecSequencePairDataSetFunction(int numPossibleLabels,
                                          boolean regression,
                                          DataVecSequencePairDataSetFunction.AlignmentMode alignmentMode,
                                          org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor,
                                          org.datavec.api.io.WritableConverter converter)

Parameters:

numPossibleLabels - Number of classes for classification (not used if regression = true)

regression - False for classification, true for regression

alignmentMode - Alignment mode for data. See DataVecSequencePairDataSetFunction.AlignmentMode

preProcessor - DataSetPreprocessor (may be null)

converter - WritableConverter (may be null)

Method Detail

call

public org.nd4j.linalg.dataset.DataSet call(scala.Tuple2<java.util.List<java.util.List<org.datavec.api.writable.Writable>>,java.util.List<java.util.List<org.datavec.api.writable.Writable>>> input)
                                     throws java.lang.Exception

Specified by:

call in interface org.apache.spark.api.java.function.Function<scala.Tuple2<java.util.List<java.util.List<org.datavec.api.writable.Writable>>,java.util.List<java.util.List<org.datavec.api.writable.Writable>>>,org.nd4j.linalg.dataset.DataSet>

Throws:

java.lang.Exception