org.deeplearning4j.spark.impl.paramavg

Class ParameterAveragingTrainingMaster

java.lang.Object

org.deeplearning4j.spark.impl.paramavg.ParameterAveragingTrainingMaster

All Implemented Interfaces:

TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

public class ParameterAveragingTrainingMaster
extends java.lang.Object
implements TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

ParameterAveragingTrainingMaster: A TrainingMaster implementation for training networks on Spark. This is standard parameter averaging with a configurable averaging period.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ParameterAveragingTrainingMaster.Builder

Constructor Summary

Constructors

Constructor and Description
ParameterAveragingTrainingMaster(boolean saveUpdater, java.lang.Integer numWorkers, int rddDataSetNumExamples, int batchSizePerWorker, int averagingFrequency, int prefetchNumBatches)
ParameterAveragingTrainingMaster(boolean saveUpdater, java.lang.Integer numWorkers, int rddDataSetNumExamples, int batchSizePerWorker, int averagingFrequency, int prefetchNumBatches, Repartition repartition, RepartitionStrategy repartitionStrategy, boolean collectTrainingStats)
ParameterAveragingTrainingMaster(boolean saveUpdater, java.lang.Integer numWorkers, int rddDataSetNumExamples, int batchSizePerWorker, int averagingFrequency, int prefetchNumBatches, Repartition repartition, RepartitionStrategy repartitionStrategy, org.apache.spark.storage.StorageLevel storageLevel, boolean collectTrainingStats)

Method Summary

Modifier and Type	Method and Description
void	addHook(TrainingHook trainingHook) Add a hook for the master for pre and post training
boolean	deleteTempFiles(org.apache.spark.api.java.JavaSparkContext sc) Attempt to delete any temporary files generated by this TrainingMaster.
boolean	deleteTempFiles(org.apache.spark.SparkContext sc) Attempt to delete any temporary files generated by this TrainingMaster.
void	executeTraining(SparkComputationGraph graph, org.apache.spark.api.java.JavaPairRDD<java.lang.String,org.apache.spark.input.PortableDataStream> trainingData) Train the SparkComputationGraph with the specified serialized DataSet objects.
void	executeTraining(SparkComputationGraph graph, org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> trainingData) Train the SparkComputationGraph with the specified data set
void	executeTraining(SparkDl4jMultiLayer network, org.apache.spark.api.java.JavaPairRDD<java.lang.String,org.apache.spark.input.PortableDataStream> trainingData) Deprecated. Due to poor performance
void	executeTraining(SparkDl4jMultiLayer network, org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> trainingData) Train the SparkDl4jMultiLayer with the specified data set
void	executeTrainingMDS(SparkComputationGraph graph, org.apache.spark.api.java.JavaPairRDD<java.lang.String,org.apache.spark.input.PortableDataStream> trainingData) Train the SparkComputationGraph with the specified serialized MultiDataSet objects.
void	executeTrainingMDS(SparkComputationGraph graph, org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> trainingData) Train the SparkComputationGraph with the specified data set
void	executeTrainingPaths(SparkComputationGraph network, org.apache.spark.api.java.JavaRDD<java.lang.String> trainingDataPaths) EXPERIMENTAL method, may be removed in a future release. Fit the network using a list of paths for serialized DataSet objects.
void	executeTrainingPaths(SparkDl4jMultiLayer network, org.apache.spark.api.java.JavaRDD<java.lang.String> trainingDataPaths) EXPERIMENTAL method, may be removed in a future release. Fit the network using a list of paths for serialized DataSet objects.
void	executeTrainingPathsMDS(SparkComputationGraph network, org.apache.spark.api.java.JavaRDD<java.lang.String> trainingMultiDataPaths) EXPERIMENTAL method, may be removed in a future release. Fit the network using a list of paths for serialized MultiDataSet objects.
static ParameterAveragingTrainingMaster	fromJson(java.lang.String jsonStr) Create a ParameterAveragingTrainingMaster instance by deserializing a JSON string that has been serialized with toJson()
static ParameterAveragingTrainingMaster	fromYaml(java.lang.String yamlStr) Create a ParameterAveragingTrainingMaster instance by deserializing a YAML string that has been serialized with toYaml()
boolean	getIsCollectTrainingStats() Get the current setting for collectTrainingStats
SparkTrainingStats	getTrainingStats() Return the training statistics.
ParameterAveragingTrainingWorker	getWorkerInstance(SparkComputationGraph graph) Get the worker instance for this training master
ParameterAveragingTrainingWorker	getWorkerInstance(SparkDl4jMultiLayer network) Get the worker instance for this training master
void	removeHook(TrainingHook trainingHook) Remove a training hook from the worker
void	setCollectTrainingStats(boolean collectTrainingStats) Set whether the training statistics should be collected.
void	setListeners(java.util.Collection<IterationListener> listeners) Set the iteration listeners.
void	setListeners(StatsStorageRouter statsStorage, java.util.Collection<IterationListener> listeners) Set the iteration listeners and the StatsStorageRouter.
java.lang.String	toJson() Get the TrainingMaster configuration as JSON
java.lang.String	toYaml() Get the TrainingMaster configuration as YAML

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParameterAveragingTrainingMaster

public ParameterAveragingTrainingMaster(boolean saveUpdater,
                                        java.lang.Integer numWorkers,
                                        int rddDataSetNumExamples,
                                        int batchSizePerWorker,
                                        int averagingFrequency,
                                        int prefetchNumBatches)

ParameterAveragingTrainingMaster

public ParameterAveragingTrainingMaster(boolean saveUpdater,
                                        java.lang.Integer numWorkers,
                                        int rddDataSetNumExamples,
                                        int batchSizePerWorker,
                                        int averagingFrequency,
                                        int prefetchNumBatches,
                                        Repartition repartition,
                                        RepartitionStrategy repartitionStrategy,
                                        boolean collectTrainingStats)

Parameters:

saveUpdater - If true: save (and average) the updater state when doing parameter averaging

numWorkers - Number of workers (executors * threads per executor) for the cluster

rddDataSetNumExamples - Number of examples in each DataSet object in the RDD<DataSet>

batchSizePerWorker - Number of examples to use per worker per fit

averagingFrequency - Frequency (in number of minibatches) with which to average parameters

prefetchNumBatches - Number of batches to asynchronously prefetch (0: disable)

collectTrainingStats - If true: collect training statistics for debugging/optimization purposes

ParameterAveragingTrainingMaster

public ParameterAveragingTrainingMaster(boolean saveUpdater,
                                        java.lang.Integer numWorkers,
                                        int rddDataSetNumExamples,
                                        int batchSizePerWorker,
                                        int averagingFrequency,
                                        int prefetchNumBatches,
                                        Repartition repartition,
                                        RepartitionStrategy repartitionStrategy,
                                        org.apache.spark.storage.StorageLevel storageLevel,
                                        boolean collectTrainingStats)

Method Detail

removeHook

public void removeHook(TrainingHook trainingHook)

Remove a training hook from the worker

Specified by:

removeHook in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

trainingHook - the training hook to remove

addHook

public void addHook(TrainingHook trainingHook)

Add a hook for the master for pre and post training

Specified by:

addHook in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

trainingHook - the training hook to add

toJson

public java.lang.String toJson()

Description copied from interface: TrainingMaster

Get the TrainingMaster configuration as JSON

Specified by:

toJson in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

toYaml

public java.lang.String toYaml()

Description copied from interface: TrainingMaster

Get the TrainingMaster configuration as YAML

Specified by:

toYaml in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

fromJson

public static ParameterAveragingTrainingMaster fromJson(java.lang.String jsonStr)

Create a ParameterAveragingTrainingMaster instance by deserializing a JSON string that has been serialized with toJson()

Parameters:

jsonStr - ParameterAveragingTrainingMaster configuration serialized as JSON

fromYaml

public static ParameterAveragingTrainingMaster fromYaml(java.lang.String yamlStr)

Create a ParameterAveragingTrainingMaster instance by deserializing a YAML string that has been serialized with toYaml()

Parameters:

yamlStr - ParameterAveragingTrainingMaster configuration serialized as YAML

getWorkerInstance

public ParameterAveragingTrainingWorker getWorkerInstance(SparkDl4jMultiLayer network)

Description copied from interface: TrainingMaster

Get the worker instance for this training master

Specified by:

getWorkerInstance in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

network - Current SparkDl4jMultiLayer

Returns:

Worker instance

getWorkerInstance

public ParameterAveragingTrainingWorker getWorkerInstance(SparkComputationGraph graph)

Description copied from interface: TrainingMaster

Get the worker instance for this training master

Specified by:

getWorkerInstance in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

graph - Current SparkComputationGraph

Returns:

Worker instance

executeTraining

public void executeTraining(SparkDl4jMultiLayer network,
                            org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> trainingData)

Description copied from interface: TrainingMaster

Train the SparkDl4jMultiLayer with the specified data set

Specified by:

executeTraining in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

network - Current network state

trainingData - Data to train on

executeTraining

@Deprecated
public void executeTraining(SparkDl4jMultiLayer network,
                                        org.apache.spark.api.java.JavaPairRDD<java.lang.String,org.apache.spark.input.PortableDataStream> trainingData)

Deprecated. Due to poor performance

Description copied from interface: TrainingMaster

Train the SparkDl4jMultiLayer with the specified serialized DataSet objects. The assumption here is that the PortableDataStreams are for DataSet objects, one per file.

Specified by:

executeTraining in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

network - Current network state

trainingData - Data to train on

executeTrainingPaths

public void executeTrainingPaths(SparkDl4jMultiLayer network,
                                 org.apache.spark.api.java.JavaRDD<java.lang.String> trainingDataPaths)

Description copied from interface: TrainingMaster

EXPERIMENTAL method, may be removed in a future release.
Fit the network using a list of paths for serialized DataSet objects.

Specified by:

executeTrainingPaths in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

network - Current network state

trainingDataPaths - Data to train on

executeTraining

public void executeTraining(SparkComputationGraph graph,
                            org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> trainingData)

Description copied from interface: TrainingMaster

Train the SparkComputationGraph with the specified data set

Specified by:

executeTraining in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

graph - Current network state

trainingData - Data to train on

executeTrainingMDS

public void executeTrainingMDS(SparkComputationGraph graph,
                               org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> trainingData)

Description copied from interface: TrainingMaster

Train the SparkComputationGraph with the specified data set

Specified by:

executeTrainingMDS in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

graph - Current network state

trainingData - Data to train on

executeTraining

public void executeTraining(SparkComputationGraph graph,
                            org.apache.spark.api.java.JavaPairRDD<java.lang.String,org.apache.spark.input.PortableDataStream> trainingData)

Description copied from interface: TrainingMaster

Train the SparkComputationGraph with the specified serialized DataSet objects. The assumption here is that the PortableDataStreams are for DataSet objects, one per file, and that these have been serialized using DataSet.save(OutputStream)

Specified by:

executeTraining in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

graph - Current network state

trainingData - Data to train on

executeTrainingMDS

public void executeTrainingMDS(SparkComputationGraph graph,
                               org.apache.spark.api.java.JavaPairRDD<java.lang.String,org.apache.spark.input.PortableDataStream> trainingData)

Description copied from interface: TrainingMaster

Train the SparkComputationGraph with the specified serialized MultiDataSet objects. The assumption here is that the PortableDataStreams are for MultiDataSet objects, one per file.

Specified by:

executeTrainingMDS in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

graph - Current network state

trainingData - Data to train on

executeTrainingPaths

public void executeTrainingPaths(SparkComputationGraph network,
                                 org.apache.spark.api.java.JavaRDD<java.lang.String> trainingDataPaths)

Description copied from interface: TrainingMaster

EXPERIMENTAL method, may be removed in a future release.
Fit the network using a list of paths for serialized DataSet objects.

Specified by:

executeTrainingPaths in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

network - Current network state

trainingDataPaths - Data to train on

executeTrainingPathsMDS

public void executeTrainingPathsMDS(SparkComputationGraph network,
                                    org.apache.spark.api.java.JavaRDD<java.lang.String> trainingMultiDataPaths)

Description copied from interface: TrainingMaster

EXPERIMENTAL method, may be removed in a future release.
Fit the network using a list of paths for serialized MultiDataSet objects.

Specified by:

executeTrainingPathsMDS in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

network - Current network state

trainingMultiDataPaths - Data to train on

setCollectTrainingStats

public void setCollectTrainingStats(boolean collectTrainingStats)

Description copied from interface: TrainingMaster

Set whether the training statistics should be collected. Training statistics may include things like per-epoch run times, time spent waiting for data, etc.

These statistics are primarily used for debugging and optimization, in order to gain some insight into what aspects of network training are taking the most time.

Specified by:

setCollectTrainingStats in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

collectTrainingStats - If true: collecting training statistics will be

getIsCollectTrainingStats

public boolean getIsCollectTrainingStats()

Description copied from interface: TrainingMaster

Get the current setting for collectTrainingStats

Specified by:

getIsCollectTrainingStats in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

getTrainingStats

public SparkTrainingStats getTrainingStats()

Description copied from interface: TrainingMaster

Return the training statistics. Note that this may return null, unless setCollectTrainingStats has been set first

Specified by:

getTrainingStats in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Returns:

Training statistics

setListeners

public void setListeners(java.util.Collection<IterationListener> listeners)

Description copied from interface: TrainingMaster

Set the iteration listeners. These should be called after every averaging (or similar) operation in the TrainingMaster, though the exact behaviour may be dependent on each IterationListener

Specified by:

setListeners in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

listeners - Listeners to set

setListeners

public void setListeners(StatsStorageRouter statsStorage,
                         java.util.Collection<IterationListener> listeners)

Description copied from interface: TrainingMaster

Set the iteration listeners and the StatsStorageRouter. This is typically used for UI functionality: for example, setListeners(new FileStatsStorage(myFile), Collections.singletonList(new StatsListener(null))). This will pass a StatsListener to each worker, and then shuffle the results back to the specified FileStatsStorage instance (which can then be attached to the UI or loaded later)

Specified by:

setListeners in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

statsStorage - StatsStorageRouter in which to place the results

listeners - Listeners

deleteTempFiles

public boolean deleteTempFiles(org.apache.spark.api.java.JavaSparkContext sc)

Description copied from interface: TrainingMaster

Attempt to delete any temporary files generated by this TrainingMaster. Depending on the configuration, no temporary files may be generated.

Specified by:

deleteTempFiles in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

sc - JavaSparkContext (used to access HDFS etc file systems, when required)

Returns:

True if deletion was successful (or, no files to delete); false otherwise.

deleteTempFiles

public boolean deleteTempFiles(org.apache.spark.SparkContext sc)

Description copied from interface: TrainingMaster

Attempt to delete any temporary files generated by this TrainingMaster. Depending on the configuration, no temporary files may be generated.

Specified by:

deleteTempFiles in interface TrainingMaster<ParameterAveragingTrainingResult,ParameterAveragingTrainingWorker>

Parameters:

sc - SparkContext (used to access HDFS etc file systems, when required)

Returns:

True if deletion was successful (or, no files to delete); false otherwise.