org.deeplearning4j.spark.util

Class MLLibUtil

java.lang.Object

org.deeplearning4j.spark.util.MLLibUtil

public class MLLibUtil
extends java.lang.Object

Dl4j <----> MLLib

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint>	fromBinary(org.apache.spark.api.java.JavaPairRDD<java.lang.String,org.apache.spark.input.PortableDataStream> binaryFiles, org.datavec.api.records.reader.RecordReader reader) Convert a traditional sc.binaryFiles in to something usable for machine learning
static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint>	fromBinary(org.apache.spark.api.java.JavaRDD<scala.Tuple2<java.lang.String,org.apache.spark.input.PortableDataStream>> binaryFiles, org.datavec.api.records.reader.RecordReader reader) Convert a traditional sc.binaryFiles in to something usable for machine learning
static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet>	fromContinuousLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data) Converts a continuous JavaRDD LabeledPoint to a JavaRDD DataSet.
static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet>	fromContinuousLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data, boolean preCache) Converts a continuous JavaRDD LabeledPoint to a JavaRDD DataSet.
static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet>	fromContinuousLabeledPoint(org.apache.spark.api.java.JavaSparkContext sc, org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data) Deprecated. Use fromContinuousLabeledPoint(JavaRDD)
static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint>	fromDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data) Convert an rdd of data set in to labeled point.
static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint>	fromDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean preCache) Convert an rdd of data set in to labeled point.
static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint>	fromDataSet(org.apache.spark.api.java.JavaSparkContext sc, org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data) Deprecated. Use fromDataSet(JavaRDD)
static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet>	fromLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data, int numPossibleLabels) Converts JavaRDD labeled points to JavaRDD datasets.
static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet>	fromLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data, int numPossibleLabels, boolean preCache) Converts JavaRDD labeled points to JavaRDD DataSets.
static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet>	fromLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data, int numPossibleLabels, int batchSize) Convert an rdd of labeled point based on the specified batch size in to data set
static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet>	fromLabeledPoint(org.apache.spark.api.java.JavaSparkContext sc, org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data, int numPossibleLabels) Deprecated. Use fromLabeledPoint(JavaRDD, int)
static org.apache.spark.mllib.regression.LabeledPoint	pointOf(java.util.Collection<org.datavec.api.writable.Writable> writables) Returns a labeled point of the writables where the final item is the point and the rest of the items are features
static double	toClassifierPrediction(org.apache.spark.mllib.linalg.Vector vector) This is for the edge case where you have a single output layer and need to convert the output layer to an index
static org.apache.spark.mllib.linalg.Matrix	toMatrix(org.nd4j.linalg.api.ndarray.INDArray arr) Convert an ndarray to a matrix.
static org.nd4j.linalg.api.ndarray.INDArray	toMatrix(org.apache.spark.mllib.linalg.Matrix arr) Convert an ndarray to a matrix.
static org.apache.spark.mllib.linalg.Vector	toVector(org.nd4j.linalg.api.ndarray.INDArray arr) Convert an ndarray to a vector
static org.nd4j.linalg.api.ndarray.INDArray	toVector(org.apache.spark.mllib.linalg.Vector arr) Convert an ndarray to a vector

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

toClassifierPrediction

public static double toClassifierPrediction(org.apache.spark.mllib.linalg.Vector vector)

This is for the edge case where you have a single output layer and need to convert the output layer to an index

Parameters:

vector - the vector to get the classifier prediction for

Returns:

the prediction for the given vector

toMatrix

public static org.nd4j.linalg.api.ndarray.INDArray toMatrix(org.apache.spark.mllib.linalg.Matrix arr)

Convert an ndarray to a matrix. Note that the matrix will be con

Parameters:

arr - the array

Returns:

an mllib vector

toVector

public static org.nd4j.linalg.api.ndarray.INDArray toVector(org.apache.spark.mllib.linalg.Vector arr)

Convert an ndarray to a vector

Parameters:

arr - the array

Returns:

an mllib vector

toMatrix

public static org.apache.spark.mllib.linalg.Matrix toMatrix(org.nd4j.linalg.api.ndarray.INDArray arr)

Convert an ndarray to a matrix. Note that the matrix will be con

Parameters:

arr - the array

Returns:

an mllib vector

toVector

public static org.apache.spark.mllib.linalg.Vector toVector(org.nd4j.linalg.api.ndarray.INDArray arr)

Convert an ndarray to a vector

Parameters:

arr - the array

Returns:

an mllib vector

fromBinary

public static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> fromBinary(org.apache.spark.api.java.JavaPairRDD<java.lang.String,org.apache.spark.input.PortableDataStream> binaryFiles,
                                                                                                           org.datavec.api.records.reader.RecordReader reader)

Convert a traditional sc.binaryFiles in to something usable for machine learning

Parameters:

binaryFiles - the binary files to convert

reader - the reader to use

Returns:

the labeled points based on the given rdd

fromBinary

public static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> fromBinary(org.apache.spark.api.java.JavaRDD<scala.Tuple2<java.lang.String,org.apache.spark.input.PortableDataStream>> binaryFiles,
                                                                                                           org.datavec.api.records.reader.RecordReader reader)

Convert a traditional sc.binaryFiles in to something usable for machine learning

Parameters:

binaryFiles - the binary files to convert

reader - the reader to use

Returns:

the labeled points based on the given rdd

pointOf

public static org.apache.spark.mllib.regression.LabeledPoint pointOf(java.util.Collection<org.datavec.api.writable.Writable> writables)

Returns a labeled point of the writables where the final item is the point and the rest of the items are features

Parameters:

writables - the writables

Returns:

the labeled point

fromLabeledPoint

public static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> fromLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data,
                                                                                                  int numPossibleLabels,
                                                                                                  int batchSize)

Convert an rdd of labeled point based on the specified batch size in to data set

Parameters:

data - the data to convert

numPossibleLabels - the number of possible labels

batchSize - the batch size

Returns:

the new rdd

fromLabeledPoint

@Deprecated
public static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> fromLabeledPoint(org.apache.spark.api.java.JavaSparkContext sc,
                                                                                                              org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data,
                                                                                                              int numPossibleLabels)

Deprecated. Use fromLabeledPoint(JavaRDD, int)

From labeled point

Parameters:

sc - the org.deeplearning4j.spark context used for creating the rdd

data - the data to convert

numPossibleLabels - the number of possible labels

Returns:

fromContinuousLabeledPoint

@Deprecated
public static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> fromContinuousLabeledPoint(org.apache.spark.api.java.JavaSparkContext sc,
                                                                                                                        org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data)

Deprecated. Use fromContinuousLabeledPoint(JavaRDD)

Convert rdd labeled points to a rdd dataset with continuous features

Parameters:

data - the java rdd labeled points ready to convert

Returns:

a JavaRDD with a continuous label

fromDataSet

@Deprecated
public static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> fromDataSet(org.apache.spark.api.java.JavaSparkContext sc,
                                                                                                                        org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data)

Deprecated. Use fromDataSet(JavaRDD)

Convert an rdd of data set in to labeled point

Parameters:

sc - the spark context to use

data - the dataset to convert

Returns:

an rdd of labeled point

fromContinuousLabeledPoint

public static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> fromContinuousLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data)

Converts a continuous JavaRDD LabeledPoint to a JavaRDD DataSet.

Parameters:

data - JavaRDD LabeledPoint

Returns:

JavaRdd DataSet

fromContinuousLabeledPoint

public static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> fromContinuousLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data,
                                                                                                            boolean preCache)

Converts a continuous JavaRDD LabeledPoint to a JavaRDD DataSet.

Parameters:

data - JavaRdd LabeledPoint

preCache - boolean pre-cache rdd before operation

Returns:

fromLabeledPoint

public static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> fromLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data,
                                                                                                  int numPossibleLabels)

Converts JavaRDD labeled points to JavaRDD datasets.

Parameters:

data - JavaRDD LabeledPoints

numPossibleLabels - number of possible labels

Returns:

fromLabeledPoint

public static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> fromLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data,
                                                                                                  int numPossibleLabels,
                                                                                                  boolean preCache)

Converts JavaRDD labeled points to JavaRDD DataSets.

Parameters:

data - JavaRDD LabeledPoints

numPossibleLabels - number of possible labels

preCache - boolean pre-cache rdd before operation

Returns:

fromDataSet

public static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> fromDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data)

Convert an rdd of data set in to labeled point.

Parameters:

data - the dataset to convert

Returns:

an rdd of labeled point

fromDataSet

public static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> fromDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                                                                                            boolean preCache)

Convert an rdd of data set in to labeled point.

Parameters:

data - the dataset to convert

preCache - boolean pre-cache rdd before operation

Returns:

an rdd of labeled point