org.deeplearning4j.spark.impl.paramavg.stats

Class ParameterAveragingTrainingMasterStats

java.lang.Object

org.deeplearning4j.spark.impl.paramavg.stats.ParameterAveragingTrainingMasterStats

All Implemented Interfaces:

java.io.Serializable, SparkTrainingStats

public class ParameterAveragingTrainingMasterStats
extends java.lang.Object
implements SparkTrainingStats

Statistics collected by a ParameterAveragingTrainingMaster

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ParameterAveragingTrainingMasterStats.ParameterAveragingTrainingMasterStatsHelper

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_DELIMITER
static java.lang.String	FILENAME_AGGREGATE_TIME
static java.lang.String	FILENAME_BROADCAST_CREATE
static java.lang.String	FILENAME_COUNT_RDD_SIZE
static java.lang.String	FILENAME_EXPORT_RDD_TIME
static java.lang.String	FILENAME_FIT_TIME
static java.lang.String	FILENAME_MAP_PARTITIONS_TIME
static java.lang.String	FILENAME_PROCESS_PARAMS_TIME
static java.lang.String	FILENAME_REPARTITION_STATS
static java.lang.String	FILENAME_SPLIT_TIME
static java.lang.String	PARAMETER_AVERAGING_MASTER_AGGREGATE_TIMES_MS
static java.lang.String	PARAMETER_AVERAGING_MASTER_BROADCAST_CREATE_TIMES_MS
static java.lang.String	PARAMETER_AVERAGING_MASTER_COUNT_RDD_TIMES_MS
static java.lang.String	PARAMETER_AVERAGING_MASTER_EXPORT_RDD_TIMES_MS
static java.lang.String	PARAMETER_AVERAGING_MASTER_FIT_TIMES_MS
static java.lang.String	PARAMETER_AVERAGING_MASTER_MAP_PARTITIONS_TIMES_MS
static java.lang.String	PARAMETER_AVERAGING_MASTER_PROCESS_PARAMS_UPDATER_TIMES_MS
static java.lang.String	PARAMETER_AVERAGING_MASTER_REPARTITION_TIMES_MS
static java.lang.String	PARAMETER_AVERAGING_MASTER_SPLIT_TIMES_MS

Fields inherited from interface org.deeplearning4j.spark.api.stats.SparkTrainingStats

DEFAULT_PRINT_FORMAT, PRINT_INDENT

Constructor Summary

Constructors

Constructor and Description

ParameterAveragingTrainingMasterStats(SparkTrainingStats workerStats, java.util.List<EventStats> parameterAveragingMasterExportTimesMs, java.util.List<EventStats> parameterAveragingMasterCountRddSizeTimesMs, java.util.List<EventStats> parameterAveragingMasterBroadcastCreateTimeMs, java.util.List<EventStats> parameterAveragingMasterFitTimeMs, java.util.List<EventStats> parameterAveragingMasterSplitTimeMs, java.util.List<EventStats> parameterAveragingMasterMapPartitionsTimesMs, java.util.List<EventStats> parameterAveragingMasterAggregateTimesMs, java.util.List<EventStats> parameterAveragingMasterProcessParamsUpdaterTimesMs, java.util.List<EventStats> parameterAveragingMasterRepartitionTimesMs)

Method Summary

Modifier and Type	Method and Description
void	addOtherTrainingStats(SparkTrainingStats other) Combine the two training stats instances.
boolean	defaultIncludeInPlots(java.lang.String key) When plotting statistics, we don't necessarily want to plot everything.
void	exportStatFiles(java.lang.String outputPath, org.apache.spark.SparkContext sc) Export the stats as a collection of files.
java.util.Set<java.lang.String>	getKeySet()
SparkTrainingStats	getNestedTrainingStats() Return the nested training stats - if any.
java.lang.String	getShortNameForKey(java.lang.String key) Return a short (display) name for the given key.
java.util.List<EventStats>	getValue(java.lang.String key) Get the statistic value for this key
java.lang.String	statsAsString() Get a String representation of the stats.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_DELIMITER

public static final java.lang.String DEFAULT_DELIMITER

See Also:

Constant Field Values

FILENAME_EXPORT_RDD_TIME

public static final java.lang.String FILENAME_EXPORT_RDD_TIME

See Also:

Constant Field Values

FILENAME_COUNT_RDD_SIZE

public static final java.lang.String FILENAME_COUNT_RDD_SIZE

See Also:

Constant Field Values

FILENAME_BROADCAST_CREATE

public static final java.lang.String FILENAME_BROADCAST_CREATE

See Also:

Constant Field Values

FILENAME_FIT_TIME

public static final java.lang.String FILENAME_FIT_TIME

See Also:

Constant Field Values

FILENAME_SPLIT_TIME

public static final java.lang.String FILENAME_SPLIT_TIME

See Also:

Constant Field Values

FILENAME_MAP_PARTITIONS_TIME

public static final java.lang.String FILENAME_MAP_PARTITIONS_TIME

See Also:

Constant Field Values

FILENAME_AGGREGATE_TIME

public static final java.lang.String FILENAME_AGGREGATE_TIME

See Also:

Constant Field Values

FILENAME_PROCESS_PARAMS_TIME

public static final java.lang.String FILENAME_PROCESS_PARAMS_TIME

See Also:

Constant Field Values

FILENAME_REPARTITION_STATS

public static final java.lang.String FILENAME_REPARTITION_STATS

See Also:

Constant Field Values

PARAMETER_AVERAGING_MASTER_EXPORT_RDD_TIMES_MS

public static final java.lang.String PARAMETER_AVERAGING_MASTER_EXPORT_RDD_TIMES_MS

See Also:

Constant Field Values

PARAMETER_AVERAGING_MASTER_COUNT_RDD_TIMES_MS

public static final java.lang.String PARAMETER_AVERAGING_MASTER_COUNT_RDD_TIMES_MS

See Also:

Constant Field Values

PARAMETER_AVERAGING_MASTER_BROADCAST_CREATE_TIMES_MS

public static final java.lang.String PARAMETER_AVERAGING_MASTER_BROADCAST_CREATE_TIMES_MS

See Also:

Constant Field Values

PARAMETER_AVERAGING_MASTER_FIT_TIMES_MS

public static final java.lang.String PARAMETER_AVERAGING_MASTER_FIT_TIMES_MS

See Also:

Constant Field Values

PARAMETER_AVERAGING_MASTER_SPLIT_TIMES_MS

public static final java.lang.String PARAMETER_AVERAGING_MASTER_SPLIT_TIMES_MS

See Also:

Constant Field Values

PARAMETER_AVERAGING_MASTER_MAP_PARTITIONS_TIMES_MS

public static final java.lang.String PARAMETER_AVERAGING_MASTER_MAP_PARTITIONS_TIMES_MS

See Also:

Constant Field Values

PARAMETER_AVERAGING_MASTER_AGGREGATE_TIMES_MS

public static final java.lang.String PARAMETER_AVERAGING_MASTER_AGGREGATE_TIMES_MS

See Also:

Constant Field Values

PARAMETER_AVERAGING_MASTER_PROCESS_PARAMS_UPDATER_TIMES_MS

public static final java.lang.String PARAMETER_AVERAGING_MASTER_PROCESS_PARAMS_UPDATER_TIMES_MS

See Also:

Constant Field Values

PARAMETER_AVERAGING_MASTER_REPARTITION_TIMES_MS

public static final java.lang.String PARAMETER_AVERAGING_MASTER_REPARTITION_TIMES_MS

See Also:

Constant Field Values

Constructor Detail

ParameterAveragingTrainingMasterStats

public ParameterAveragingTrainingMasterStats(SparkTrainingStats workerStats,
                                             java.util.List<EventStats> parameterAveragingMasterExportTimesMs,
                                             java.util.List<EventStats> parameterAveragingMasterCountRddSizeTimesMs,
                                             java.util.List<EventStats> parameterAveragingMasterBroadcastCreateTimeMs,
                                             java.util.List<EventStats> parameterAveragingMasterFitTimeMs,
                                             java.util.List<EventStats> parameterAveragingMasterSplitTimeMs,
                                             java.util.List<EventStats> parameterAveragingMasterMapPartitionsTimesMs,
                                             java.util.List<EventStats> parameterAveragingMasterAggregateTimesMs,
                                             java.util.List<EventStats> parameterAveragingMasterProcessParamsUpdaterTimesMs,
                                             java.util.List<EventStats> parameterAveragingMasterRepartitionTimesMs)

Method Detail

getKeySet

public java.util.Set<java.lang.String> getKeySet()

Specified by:

getKeySet in interface SparkTrainingStats

Returns:

Set of keys that can be used with SparkTrainingStats.getValue(String)

getValue

public java.util.List<EventStats> getValue(java.lang.String key)

Description copied from interface: SparkTrainingStats

Get the statistic value for this key

Specified by:

getValue in interface SparkTrainingStats

Parameters:

key - Key to get the value for

Returns:

Statistic for this key, or an exception if key is invalid

getShortNameForKey

public java.lang.String getShortNameForKey(java.lang.String key)

Description copied from interface: SparkTrainingStats

Return a short (display) name for the given key.

Specified by:

getShortNameForKey in interface SparkTrainingStats

Parameters:

key - Key

Returns:

Short/display name for key

defaultIncludeInPlots

public boolean defaultIncludeInPlots(java.lang.String key)

Description copied from interface: SparkTrainingStats

When plotting statistics, we don't necessarily want to plot everything. For example, some statistics/measurements are made up multiple smaller components; it does not always make sense to plot both the larger stat, and the components that make it up

Specified by:

defaultIncludeInPlots in interface SparkTrainingStats

Parameters:

key - Key to check for default plotting behaviour

Returns:

Whether the specified key should be included in plots by default or not

addOtherTrainingStats

public void addOtherTrainingStats(SparkTrainingStats other)

Description copied from interface: SparkTrainingStats

Combine the two training stats instances. Usually, the two objects must be of the same type

Specified by:

addOtherTrainingStats in interface SparkTrainingStats

Parameters:

other - Other training stats to return

getNestedTrainingStats

public SparkTrainingStats getNestedTrainingStats()

Description copied from interface: SparkTrainingStats

Return the nested training stats - if any.

Specified by:

getNestedTrainingStats in interface SparkTrainingStats

Returns:

The nested stats, if present/applicable, or null otherwise

statsAsString

public java.lang.String statsAsString()

Description copied from interface: SparkTrainingStats

Get a String representation of the stats. This functionality is implemented as a separate method (as opposed to toString()) as the resulting String can be very large.
NOTE: The String representation typically includes only duration information. To get full statistics (including machine IDs, etc) use SparkTrainingStats.getValue(String) or export full data via SparkTrainingStats.exportStatFiles(String, SparkContext)

Specified by:

statsAsString in interface SparkTrainingStats

Returns:

A String representation of the training statistics

exportStatFiles

public void exportStatFiles(java.lang.String outputPath,
                            org.apache.spark.SparkContext sc)
                     throws java.io.IOException

Description copied from interface: SparkTrainingStats

Export the stats as a collection of files. Stats are comma-delimited (CSV) with 1 header line

Specified by:

exportStatFiles in interface SparkTrainingStats

Parameters:

outputPath - Base directory to write files to

Throws:

java.io.IOException