org.deeplearning4j.spark.api.stats

Interface SparkTrainingStats

All Superinterfaces:

java.io.Serializable

All Known Implementing Classes:

CommonSparkTrainingStats, ParameterAveragingTrainingMasterStats, ParameterAveragingTrainingWorkerStats

public interface SparkTrainingStats
extends java.io.Serializable

SparkTrainingStats is an interface that is used for accessing training statistics, for multiple TrainingMaster implementations.

The idea is that for debugging purposes, we want to collect a number of statistics related to the training. However, these statistics will vary, depending on which the type of training we are doing. Specifically, both the keys (number/names of stats) and their actual values (types/classes) can vary.

The interface here operates essentially as a Map<String,Object>. Note however that SparkTrainingStats instances may be nested: for example a ParameterAveragingTrainingMasterStats may have a CommonSparkTrainingStats instance which may in turn have a ParameterAveragingTrainingWorkerStats instance.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_PRINT_FORMAT Default formatter used for statsAsString()
static int	PRINT_INDENT Default indentation for statsAsString()

Method Summary

Modifier and Type	Method and Description
void	addOtherTrainingStats(SparkTrainingStats other) Combine the two training stats instances.
boolean	defaultIncludeInPlots(java.lang.String key) When plotting statistics, we don't necessarily want to plot everything.
void	exportStatFiles(java.lang.String outputPath, org.apache.spark.SparkContext sc) Export the stats as a collection of files.
java.util.Set<java.lang.String>	getKeySet()
SparkTrainingStats	getNestedTrainingStats() Return the nested training stats - if any.
java.lang.String	getShortNameForKey(java.lang.String key) Return a short (display) name for the given key.
java.util.List<EventStats>	getValue(java.lang.String key) Get the statistic value for this key
java.lang.String	statsAsString() Get a String representation of the stats.

Field Detail

PRINT_INDENT

static final int PRINT_INDENT

Default indentation for statsAsString()

See Also:

Constant Field Values

DEFAULT_PRINT_FORMAT

static final java.lang.String DEFAULT_PRINT_FORMAT

Default formatter used for statsAsString()

See Also:

Constant Field Values

Method Detail

getKeySet

java.util.Set<java.lang.String> getKeySet()

Returns:

Set of keys that can be used with getValue(String)

getValue

java.util.List<EventStats> getValue(java.lang.String key)

Get the statistic value for this key

Parameters:

key - Key to get the value for

Returns:

Statistic for this key, or an exception if key is invalid

getShortNameForKey

java.lang.String getShortNameForKey(java.lang.String key)

Return a short (display) name for the given key.

Parameters:

key - Key

Returns:

Short/display name for key

defaultIncludeInPlots

boolean defaultIncludeInPlots(java.lang.String key)

When plotting statistics, we don't necessarily want to plot everything. For example, some statistics/measurements are made up multiple smaller components; it does not always make sense to plot both the larger stat, and the components that make it up

Parameters:

key - Key to check for default plotting behaviour

Returns:

Whether the specified key should be included in plots by default or not

addOtherTrainingStats

void addOtherTrainingStats(SparkTrainingStats other)

Combine the two training stats instances. Usually, the two objects must be of the same type

Parameters:

other - Other training stats to return

getNestedTrainingStats

SparkTrainingStats getNestedTrainingStats()

Return the nested training stats - if any.

Returns:

The nested stats, if present/applicable, or null otherwise

statsAsString

java.lang.String statsAsString()

Get a String representation of the stats. This functionality is implemented as a separate method (as opposed to toString()) as the resulting String can be very large.
NOTE: The String representation typically includes only duration information. To get full statistics (including machine IDs, etc) use getValue(String) or export full data via exportStatFiles(String, SparkContext)

Returns:

A String representation of the training statistics

exportStatFiles

void exportStatFiles(java.lang.String outputPath,
                     org.apache.spark.SparkContext sc)
              throws java.io.IOException

Export the stats as a collection of files. Stats are comma-delimited (CSV) with 1 header line

Parameters:

outputPath - Base directory to write files to

Throws:

java.io.IOException