org.deeplearning4j.spark.api.stats

Class CommonSparkTrainingStats

java.lang.Object

org.deeplearning4j.spark.api.stats.CommonSparkTrainingStats

All Implemented Interfaces:

java.io.Serializable, SparkTrainingStats

public class CommonSparkTrainingStats
extends java.lang.Object
implements SparkTrainingStats

A SparkTrainingStats implementation for common stats functionality used by most workers

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CommonSparkTrainingStats.Builder

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_DELIMITER
static java.lang.String	FILENAME_DATASET_GET_TIME_STATS
static java.lang.String	FILENAME_GET_INITIAL_MODEL_STATS
static java.lang.String	FILENAME_PROCESS_MINIBATCH_TIME_STATS
static java.lang.String	FILENAME_TOTAL_TIME_STATS
static java.lang.String	WORKER_FLAT_MAP_DATA_SET_GET_TIMES_MS
static java.lang.String	WORKER_FLAT_MAP_GET_INITIAL_MODEL_TIME_MS
static java.lang.String	WORKER_FLAT_MAP_PROCESS_MINI_BATCH_TIMES_MS
static java.lang.String	WORKER_FLAT_MAP_TOTAL_TIME_MS

Fields inherited from interface org.deeplearning4j.spark.api.stats.SparkTrainingStats

DEFAULT_PRINT_FORMAT, PRINT_INDENT

Constructor Summary

Constructors

Constructor and Description
CommonSparkTrainingStats()

Method Summary

Modifier and Type	Method and Description
void	addOtherTrainingStats(SparkTrainingStats other) Combine the two training stats instances.
boolean	defaultIncludeInPlots(java.lang.String key) When plotting statistics, we don't necessarily want to plot everything.
void	exportStatFiles(java.lang.String outputPath, org.apache.spark.SparkContext sc) Export the stats as a collection of files.
java.util.Set<java.lang.String>	getKeySet()
SparkTrainingStats	getNestedTrainingStats() Return the nested training stats - if any.
java.lang.String	getShortNameForKey(java.lang.String key) Return a short (display) name for the given key.
java.util.List<EventStats>	getValue(java.lang.String key) Get the statistic value for this key
java.lang.String	statsAsString() Get a String representation of the stats.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_DELIMITER

public static final java.lang.String DEFAULT_DELIMITER

See Also:

Constant Field Values

FILENAME_TOTAL_TIME_STATS

public static final java.lang.String FILENAME_TOTAL_TIME_STATS

See Also:

Constant Field Values

FILENAME_GET_INITIAL_MODEL_STATS

public static final java.lang.String FILENAME_GET_INITIAL_MODEL_STATS

See Also:

Constant Field Values

FILENAME_DATASET_GET_TIME_STATS

public static final java.lang.String FILENAME_DATASET_GET_TIME_STATS

See Also:

Constant Field Values

FILENAME_PROCESS_MINIBATCH_TIME_STATS

public static final java.lang.String FILENAME_PROCESS_MINIBATCH_TIME_STATS

See Also:

Constant Field Values

WORKER_FLAT_MAP_TOTAL_TIME_MS

public static final java.lang.String WORKER_FLAT_MAP_TOTAL_TIME_MS

See Also:

Constant Field Values

WORKER_FLAT_MAP_GET_INITIAL_MODEL_TIME_MS

public static final java.lang.String WORKER_FLAT_MAP_GET_INITIAL_MODEL_TIME_MS

See Also:

Constant Field Values

WORKER_FLAT_MAP_DATA_SET_GET_TIMES_MS

public static final java.lang.String WORKER_FLAT_MAP_DATA_SET_GET_TIMES_MS

See Also:

Constant Field Values

WORKER_FLAT_MAP_PROCESS_MINI_BATCH_TIMES_MS

public static final java.lang.String WORKER_FLAT_MAP_PROCESS_MINI_BATCH_TIMES_MS

See Also:

Constant Field Values

Constructor Detail

CommonSparkTrainingStats

public CommonSparkTrainingStats()

Method Detail

getKeySet

public java.util.Set<java.lang.String> getKeySet()

Specified by:

getKeySet in interface SparkTrainingStats

Returns:

Set of keys that can be used with SparkTrainingStats.getValue(String)

getValue

public java.util.List<EventStats> getValue(java.lang.String key)

Description copied from interface: SparkTrainingStats

Get the statistic value for this key

Specified by:

getValue in interface SparkTrainingStats

Parameters:

key - Key to get the value for

Returns:

Statistic for this key, or an exception if key is invalid

getShortNameForKey

public java.lang.String getShortNameForKey(java.lang.String key)

Description copied from interface: SparkTrainingStats

Return a short (display) name for the given key.

Specified by:

getShortNameForKey in interface SparkTrainingStats

Parameters:

key - Key

Returns:

Short/display name for key

defaultIncludeInPlots

public boolean defaultIncludeInPlots(java.lang.String key)

Description copied from interface: SparkTrainingStats

When plotting statistics, we don't necessarily want to plot everything. For example, some statistics/measurements are made up multiple smaller components; it does not always make sense to plot both the larger stat, and the components that make it up

Specified by:

defaultIncludeInPlots in interface SparkTrainingStats

Parameters:

key - Key to check for default plotting behaviour

Returns:

Whether the specified key should be included in plots by default or not

addOtherTrainingStats

public void addOtherTrainingStats(SparkTrainingStats other)

Description copied from interface: SparkTrainingStats

Combine the two training stats instances. Usually, the two objects must be of the same type

Specified by:

addOtherTrainingStats in interface SparkTrainingStats

Parameters:

other - Other training stats to return

getNestedTrainingStats

public SparkTrainingStats getNestedTrainingStats()

Description copied from interface: SparkTrainingStats

Return the nested training stats - if any.

Specified by:

getNestedTrainingStats in interface SparkTrainingStats

Returns:

The nested stats, if present/applicable, or null otherwise

statsAsString

public java.lang.String statsAsString()

Description copied from interface: SparkTrainingStats

Get a String representation of the stats. This functionality is implemented as a separate method (as opposed to toString()) as the resulting String can be very large.
NOTE: The String representation typically includes only duration information. To get full statistics (including machine IDs, etc) use SparkTrainingStats.getValue(String) or export full data via SparkTrainingStats.exportStatFiles(String, SparkContext)

Specified by:

statsAsString in interface SparkTrainingStats

Returns:

A String representation of the training statistics

exportStatFiles

public void exportStatFiles(java.lang.String outputPath,
                            org.apache.spark.SparkContext sc)
                     throws java.io.IOException

Description copied from interface: SparkTrainingStats

Export the stats as a collection of files. Stats are comma-delimited (CSV) with 1 header line

Specified by:

exportStatFiles in interface SparkTrainingStats

Parameters:

outputPath - Base directory to write files to

Throws:

java.io.IOException