org.deeplearning4j.spark.impl.multilayer

Class SparkDl4jMultiLayer

java.lang.Object

org.deeplearning4j.spark.impl.SparkListenable

org.deeplearning4j.spark.impl.multilayer.SparkDl4jMultiLayer

public class SparkDl4jMultiLayer
extends SparkListenable

Master class for spark

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_EVAL_SCORE_BATCH_SIZE
static int	DEFAULT_ROC_THRESHOLD_STEPS

Fields inherited from class org.deeplearning4j.spark.impl.SparkListenable

trainingMaster

Constructor Summary

Constructors

Constructor and Description
SparkDl4jMultiLayer(org.apache.spark.api.java.JavaSparkContext sc, MultiLayerConfiguration conf, TrainingMaster<?,?> trainingMaster) Training constructor.
SparkDl4jMultiLayer(org.apache.spark.api.java.JavaSparkContext javaSparkContext, MultiLayerNetwork network, TrainingMaster<?,?> trainingMaster)
SparkDl4jMultiLayer(org.apache.spark.SparkContext sparkContext, MultiLayerConfiguration conf, TrainingMaster<?,?> trainingMaster) Training constructor.
SparkDl4jMultiLayer(org.apache.spark.SparkContext sparkContext, MultiLayerNetwork network, TrainingMaster<?,?> trainingMaster) Instantiate a multi layer spark instance with the given context and network.

Method Summary

Modifier and Type	Method and Description
double	calculateScore(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean average) Calculate the score for all examples in the provided JavaRDD<DataSet>, either by summing or averaging over the entire data set.
double	calculateScore(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean average, int minibatchSize) Calculate the score for all examples in the provided JavaRDD<DataSet>, either by summing or averaging over the entire data set.
double	calculateScore(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> data, boolean average) Overload of calculateScore(JavaRDD, boolean) for RDD<DataSet> instead of JavaRDD<DataSet>
<T extends IEvaluation> T	doEvaluation(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, T emptyEvaluation, int evalBatchSize) Perform distributed evaluation of any type of IEvaluation.
Evaluation	evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data) Evaluate the network (classification performance) in a distributed manner on the provided data
Evaluation	evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, java.util.List<java.lang.String> labelsList) Evaluate the network (classification performance) in a distributed manner, using default batch size and a provided list of labels
Evaluation	evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, java.util.List<java.lang.String> labelsList, int evalBatchSize) Evaluate the network (classification performance) in a distributed manner, using specified batch size and a provided list of labels
Evaluation	evaluate(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> data) RDD<DataSet> overload of evaluate(JavaRDD)
Evaluation	evaluate(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> data, java.util.List<java.lang.String> labelsList) RDD<DataSet> overload of evaluate(JavaRDD, List)
RegressionEvaluation	evaluateRegression(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data) Evaluate the network (regression performance) in a distributed manner on the provided data
RegressionEvaluation	evaluateRegression(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, int minibatchSize) Evaluate the network (regression performance) in a distributed manner on the provided data
ROC	evaluateROC(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data) Perform ROC analysis/evaluation on the given DataSet in a distributed manner, using the default number of threshold steps (DEFAULT_ROC_THRESHOLD_STEPS) and the default minibatch size (DEFAULT_EVAL_SCORE_BATCH_SIZE)
ROC	evaluateROC(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, int thresholdSteps, int evaluationMinibatchSize) Perform ROC analysis/evaluation on the given DataSet in a distributed manner
ROCMultiClass	evaluateROCMultiClass(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data) Perform ROC analysis/evaluation (for the multi-class case, using ROCMultiClass on the given DataSet in a distributed manner
ROCMultiClass	evaluateROCMultiClass(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, int thresholdSteps, int evaluationMinibatchSize) Perform ROC analysis/evaluation (for the multi-class case, using ROCMultiClass on the given DataSet in a distributed manner
<K> org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.api.ndarray.INDArray>	feedForwardWithKey(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.api.ndarray.INDArray> featuresData, int batchSize) Feed-forward the specified data, with the given keys.
MultiLayerNetwork	fit(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> trainingData) Fit the DataSet RDD
MultiLayerNetwork	fit(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> trainingData) Fit the DataSet RDD.
MultiLayerNetwork	fit(java.lang.String path) Fit the SparkDl4jMultiLayer network using a directory of serialized DataSet objects The assumption here is that the directory contains a number of DataSet objects, each serialized using DataSet.save(OutputStream)
MultiLayerNetwork	fit(java.lang.String path, int minPartitions) Deprecated. Use fit(String)
MultiLayerNetwork	fitContinuousLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> rdd) Fits a MultiLayerNetwork using Spark MLLib LabeledPoint instances This will convert labeled points that have continuous labels used for regression to the internal DL4J data format and train the model on that
MultiLayerNetwork	fitLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> rdd) Fit a MultiLayerNetwork using Spark MLLib LabeledPoint instances.
MultiLayerNetwork	fitPaths(org.apache.spark.api.java.JavaRDD<java.lang.String> paths) Fit the network using a list of paths for serialized DataSet objects.
MultiLayerNetwork	getNetwork()
double	getScore() Gets the last (average) minibatch score from calling fit.
org.apache.spark.api.java.JavaSparkContext	getSparkContext()
SparkTrainingStats	getSparkTrainingStats() Get the training statistics, after collection of stats has been enabled using setCollectTrainingStats(boolean)
TrainingMaster	getTrainingMaster()
org.apache.spark.mllib.linalg.Matrix	predict(org.apache.spark.mllib.linalg.Matrix features) Predict the given feature matrix
org.apache.spark.mllib.linalg.Vector	predict(org.apache.spark.mllib.linalg.Vector point) Predict the given vector
<K> org.apache.spark.api.java.JavaPairRDD<K,java.lang.Double>	scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms) Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE.
<K> org.apache.spark.api.java.JavaPairRDD<K,java.lang.Double>	scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms, int batchSize) Score the examples individually, using a specified batch size.
org.apache.spark.api.java.JavaDoubleRDD	scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms) Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE.
org.apache.spark.api.java.JavaDoubleRDD	scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms, int batchSize) Score the examples individually, using a specified batch size.
org.apache.spark.api.java.JavaDoubleRDD	scoreExamples(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms) RDD<DataSet> overload of scoreExamples(JavaPairRDD, boolean)
org.apache.spark.api.java.JavaDoubleRDD	scoreExamples(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms, int batchSize) RDD<DataSet> overload of scoreExamples(JavaRDD, boolean, int)
void	setCollectTrainingStats(boolean collectTrainingStats) Set whether training statistics should be collected for debugging purposes.
void	setNetwork(MultiLayerNetwork network) Set the network that underlies this SparkDl4jMultiLayer instacne
void	setScore(double lastScore)

Methods inherited from class org.deeplearning4j.spark.impl.SparkListenable

setListeners, setListeners, setListeners, setListeners

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_EVAL_SCORE_BATCH_SIZE

public static final int DEFAULT_EVAL_SCORE_BATCH_SIZE

See Also:

Constant Field Values

DEFAULT_ROC_THRESHOLD_STEPS

public static final int DEFAULT_ROC_THRESHOLD_STEPS

See Also:

Constant Field Values

Constructor Detail

SparkDl4jMultiLayer

public SparkDl4jMultiLayer(org.apache.spark.SparkContext sparkContext,
                           MultiLayerNetwork network,
                           TrainingMaster<?,?> trainingMaster)

Instantiate a multi layer spark instance with the given context and network. This is the prediction constructor

Parameters:

sparkContext - the spark context to use

network - the network to use

SparkDl4jMultiLayer

public SparkDl4jMultiLayer(org.apache.spark.SparkContext sparkContext,
                           MultiLayerConfiguration conf,
                           TrainingMaster<?,?> trainingMaster)

Training constructor. Instantiate with a configuration

Parameters:

sparkContext - the spark context to use

conf - the configuration of the network

SparkDl4jMultiLayer

public SparkDl4jMultiLayer(org.apache.spark.api.java.JavaSparkContext sc,
                           MultiLayerConfiguration conf,
                           TrainingMaster<?,?> trainingMaster)

Training constructor. Instantiate with a configuration

Parameters:

sc - the spark context to use

conf - the configuration of the network

SparkDl4jMultiLayer

public SparkDl4jMultiLayer(org.apache.spark.api.java.JavaSparkContext javaSparkContext,
                           MultiLayerNetwork network,
                           TrainingMaster<?,?> trainingMaster)

Method Detail

getSparkContext

public org.apache.spark.api.java.JavaSparkContext getSparkContext()

getNetwork

public MultiLayerNetwork getNetwork()

Returns:

The MultiLayerNetwork underlying the SparkDl4jMultiLayer

getTrainingMaster

public TrainingMaster getTrainingMaster()

Returns:

The TrainingMaster for this network

setNetwork

public void setNetwork(MultiLayerNetwork network)

Set the network that underlies this SparkDl4jMultiLayer instacne

Parameters:

network - network to set

setCollectTrainingStats

public void setCollectTrainingStats(boolean collectTrainingStats)

Set whether training statistics should be collected for debugging purposes. Statistics collection is disabled by default

Parameters:

collectTrainingStats - If true: collect training statistics. If false: don't collect.

getSparkTrainingStats

public SparkTrainingStats getSparkTrainingStats()

Get the training statistics, after collection of stats has been enabled using setCollectTrainingStats(boolean)

Returns:

Training statistics

predict

public org.apache.spark.mllib.linalg.Matrix predict(org.apache.spark.mllib.linalg.Matrix features)

Predict the given feature matrix

Parameters:

features - the given feature matrix

Returns:

the predictions

predict

public org.apache.spark.mllib.linalg.Vector predict(org.apache.spark.mllib.linalg.Vector point)

Predict the given vector

Parameters:

point - the vector to predict

Returns:

the predicted vector

fit

public MultiLayerNetwork fit(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> trainingData)

Fit the DataSet RDD. Equivalent to fit(trainingData.toJavaRDD())

Parameters:

trainingData - the training data RDD to fitDataSet

Returns:

the MultiLayerNetwork after training

fit

public MultiLayerNetwork fit(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> trainingData)

Fit the DataSet RDD

Parameters:

trainingData - the training data RDD to fitDataSet

Returns:

the MultiLayerNetwork after training

fit

public MultiLayerNetwork fit(java.lang.String path)

Fit the SparkDl4jMultiLayer network using a directory of serialized DataSet objects The assumption here is that the directory contains a number of DataSet objects, each serialized using DataSet.save(OutputStream)

Parameters:

path - Path to the directory containing the serialized DataSet objcets

Returns:

The MultiLayerNetwork after training

fit

@Deprecated
public MultiLayerNetwork fit(java.lang.String path,
                                         int minPartitions)

Deprecated. Use fit(String)

fitPaths

public MultiLayerNetwork fitPaths(org.apache.spark.api.java.JavaRDD<java.lang.String> paths)

Fit the network using a list of paths for serialized DataSet objects.

Parameters:

paths - List of paths

Returns:

trained network

fitLabeledPoint

public MultiLayerNetwork fitLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> rdd)

Fit a MultiLayerNetwork using Spark MLLib LabeledPoint instances. This will convert the labeled points to the internal DL4J data format and train the model on that

Parameters:

rdd - the rdd to fitDataSet

Returns:

the multi layer network that was fitDataSet

fitContinuousLabeledPoint

public MultiLayerNetwork fitContinuousLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> rdd)

Fits a MultiLayerNetwork using Spark MLLib LabeledPoint instances This will convert labeled points that have continuous labels used for regression to the internal DL4J data format and train the model on that

Parameters:

rdd - the javaRDD containing the labeled points

Returns:

a MultiLayerNetwork

getScore

public double getScore()

Gets the last (average) minibatch score from calling fit. This is the average score across all executors for the last minibatch executed in each worker

setScore

public void setScore(double lastScore)

calculateScore

public double calculateScore(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> data,
                             boolean average)

Overload of calculateScore(JavaRDD, boolean) for RDD<DataSet> instead of JavaRDD<DataSet>

calculateScore

public double calculateScore(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                             boolean average)

Calculate the score for all examples in the provided JavaRDD<DataSet>, either by summing or averaging over the entire data set. To calculate a score for each example individually, use scoreExamples(JavaPairRDD, boolean) or one of the similar methods. Uses default minibatch size in each worker, DEFAULT_EVAL_SCORE_BATCH_SIZE

Parameters:

data - Data to score

average - Whether to sum the scores, or average them

calculateScore

public double calculateScore(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                             boolean average,
                             int minibatchSize)

Calculate the score for all examples in the provided JavaRDD<DataSet>, either by summing or averaging over the entire data set. To calculate a score for each example individually, use scoreExamples(JavaPairRDD, boolean) or one of the similar methods

Parameters:

data - Data to score

average - Whether to sum the scores, or average them

minibatchSize - The number of examples to use in each minibatch when scoring. If more examples are in a partition than this, multiple scoring operations will be done (to avoid using too much memory by doing the whole partition in one go)

scoreExamples

public org.apache.spark.api.java.JavaDoubleRDD scoreExamples(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> data,
                                                             boolean includeRegularizationTerms)

RDD<DataSet> overload of scoreExamples(JavaPairRDD, boolean)

scoreExamples

public org.apache.spark.api.java.JavaDoubleRDD scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                                             boolean includeRegularizationTerms)

Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately. If scoring is needed for specific examples use either scoreExamples(JavaPairRDD, boolean) or scoreExamples(JavaPairRDD, boolean, int) which can have a key for each example.

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

Returns:

A JavaDoubleRDD containing the scores of each example

See Also:

MultiLayerNetwork.scoreExamples(DataSet, boolean)

scoreExamples

public org.apache.spark.api.java.JavaDoubleRDD scoreExamples(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> data,
                                                             boolean includeRegularizationTerms,
                                                             int batchSize)

RDD<DataSet> overload of scoreExamples(JavaRDD, boolean, int)

scoreExamples

public org.apache.spark.api.java.JavaDoubleRDD scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                                             boolean includeRegularizationTerms,
                                                             int batchSize)

Score the examples individually, using a specified batch size. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately. If scoring is needed for specific examples use either scoreExamples(JavaPairRDD, boolean) or scoreExamples(JavaPairRDD, boolean, int) which can have a key for each example.

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

batchSize - Batch size to use when doing scoring

Returns:

A JavaDoubleRDD containing the scores of each example

See Also:

MultiLayerNetwork.scoreExamples(DataSet, boolean)

scoreExamples

public <K> org.apache.spark.api.java.JavaPairRDD<K,java.lang.Double> scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data,
                                                                                   boolean includeRegularizationTerms)

Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately
Note: The provided JavaPairRDD has a key that is associated with each example and returned score.
Note: The DataSet objects passed in must have exactly one example in them (otherwise: can't have a 1:1 association between keys and data sets to score)

Type Parameters:

K - Key type

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

Returns:

A JavaPairRDD<K,Double> containing the scores of each example

See Also:

MultiLayerNetwork.scoreExamples(DataSet, boolean)

scoreExamples

public <K> org.apache.spark.api.java.JavaPairRDD<K,java.lang.Double> scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data,
                                                                                   boolean includeRegularizationTerms,
                                                                                   int batchSize)

Score the examples individually, using a specified batch size. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately
Note: The provided JavaPairRDD has a key that is associated with each example and returned score.
Note: The DataSet objects passed in must have exactly one example in them (otherwise: can't have a 1:1 association between keys and data sets to score)

Type Parameters:

K - Key type

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

Returns:

A JavaPairRDD<K,Double> containing the scores of each example

See Also:

MultiLayerNetwork.scoreExamples(DataSet, boolean)

feedForwardWithKey

public <K> org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.api.ndarray.INDArray> feedForwardWithKey(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.api.ndarray.INDArray> featuresData,
                                                                                                            int batchSize)

Feed-forward the specified data, with the given keys. i.e., get the network output/predictions for the specified data

Type Parameters:

K - Type of data for key - may be anything

Parameters:

featuresData - Features data to feed through the network

batchSize - Batch size to use when doing feed forward operations

Returns:

Network output given the input, by key

evaluate

public Evaluation evaluate(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> data)

RDD<DataSet> overload of evaluate(JavaRDD)

evaluate

public Evaluation evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data)

Evaluate the network (classification performance) in a distributed manner on the provided data

Parameters:

data - Data to evaluate on

Returns:

Evaluation object; results of evaluation on all examples in the data set

evaluate

public Evaluation evaluate(org.apache.spark.rdd.RDD<org.nd4j.linalg.dataset.DataSet> data,
                           java.util.List<java.lang.String> labelsList)

RDD<DataSet> overload of evaluate(JavaRDD, List)

evaluateRegression

public RegressionEvaluation evaluateRegression(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data)

Evaluate the network (regression performance) in a distributed manner on the provided data

Parameters:

data - Data to evaluate

Returns:

RegressionEvaluation instance with regression performance

evaluateRegression

public RegressionEvaluation evaluateRegression(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                               int minibatchSize)

Evaluate the network (regression performance) in a distributed manner on the provided data

Parameters:

data - Data to evaluate

minibatchSize - Minibatch size to use when doing performing evaluation

Returns:

RegressionEvaluation instance with regression performance

evaluate

public Evaluation evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                           java.util.List<java.lang.String> labelsList)

Evaluate the network (classification performance) in a distributed manner, using default batch size and a provided list of labels

Parameters:

data - Data to evaluate on

labelsList - List of labels used for evaluation

Returns:

Evaluation object; results of evaluation on all examples in the data set

evaluateROC

public ROC evaluateROC(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data)

Perform ROC analysis/evaluation on the given DataSet in a distributed manner, using the default number of threshold steps (DEFAULT_ROC_THRESHOLD_STEPS) and the default minibatch size (DEFAULT_EVAL_SCORE_BATCH_SIZE)

Parameters:

data - Test set data (to evaluate on)

Returns:

ROC for the entire data set

evaluateROC

public ROC evaluateROC(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                       int thresholdSteps,
                       int evaluationMinibatchSize)

Perform ROC analysis/evaluation on the given DataSet in a distributed manner

Parameters:

data - Test set data (to evaluate on)

thresholdSteps - Number of threshold steps for ROC - see ROC

evaluationMinibatchSize - Minibatch size to use when performing ROC evaluation

Returns:

ROC for the entire data set

evaluateROCMultiClass

public ROCMultiClass evaluateROCMultiClass(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data)

Perform ROC analysis/evaluation (for the multi-class case, using ROCMultiClass on the given DataSet in a distributed manner

Parameters:

data - Test set data (to evaluate on)

Returns:

ROC for the entire data set

evaluateROCMultiClass

public ROCMultiClass evaluateROCMultiClass(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                           int thresholdSteps,
                                           int evaluationMinibatchSize)

Perform ROC analysis/evaluation (for the multi-class case, using ROCMultiClass on the given DataSet in a distributed manner

Parameters:

data - Test set data (to evaluate on)

thresholdSteps - Number of threshold steps for ROC - see ROC

evaluationMinibatchSize - Minibatch size to use when performing ROC evaluation

Returns:

ROCMultiClass for the entire data set

evaluate

public Evaluation evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                           java.util.List<java.lang.String> labelsList,
                           int evalBatchSize)

Evaluate the network (classification performance) in a distributed manner, using specified batch size and a provided list of labels

Parameters:

data - Data to evaluate on

labelsList - List of labels used for evaluation

evalBatchSize - Batch size to use when conducting evaluations

Returns:

Evaluation object; results of evaluation on all examples in the data set

doEvaluation

public <T extends IEvaluation> T doEvaluation(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                              T emptyEvaluation,
                                              int evalBatchSize)

Perform distributed evaluation of any type of IEvaluation. For example, Evaluation, RegressionEvaluation, ROC, ROCMultiClass etc.

Type Parameters:

T - Type of evaluation instance to return

Parameters:

data - Data to evaluate on

emptyEvaluation - Empty evaluation instance. This is the starting point (serialized/duplicated, then merged)

evalBatchSize - Evaluation batch size

Returns:

IEvaluation instance