org.deeplearning4j.spark.stats

Class StatsUtils

java.lang.Object

org.deeplearning4j.spark.stats.StatsUtils

public class StatsUtils
extends java.lang.Object

Utility methods for Spark training stats

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StatsUtils.StartTimeComparator

Field Summary

Fields

Modifier and Type	Field and Description
static long	DEFAULT_MAX_TIMELINE_SIZE_MS

Method Summary

Modifier and Type	Method and Description
static void	exportStats(java.util.List<EventStats> list, java.lang.String outputPath, java.lang.String delimiter, org.apache.spark.SparkContext sc)
static void	exportStats(java.util.List<EventStats> list, java.lang.String outputDirectory, java.lang.String filename, java.lang.String delimiter, org.apache.spark.SparkContext sc)
static void	exportStatsAsHTML(SparkTrainingStats sparkTrainingStats, long maxTimelineSizeMs, java.io.OutputStream outputStream) Generate and export a HTML representation (including charts, etc) of the Spark training statistics This overload is for writing to an output stream
static void	exportStatsAsHtml(SparkTrainingStats sparkTrainingStats, long maxTimelineSizeMs, java.lang.String path, org.apache.spark.SparkContext sc) Generate and export a HTML representation (including charts, etc) of the Spark training statistics Note: exporting is done via Spark, so the path here can be a local file, HDFS, etc.
static void	exportStatsAsHTML(SparkTrainingStats sparkTrainingStats, java.io.OutputStream outputStream) Generate and export a HTML representation (including charts, etc) of the Spark training statistics This overload is for writing to an output stream
static void	exportStatsAsHtml(SparkTrainingStats sparkTrainingStats, java.lang.String path, org.apache.spark.api.java.JavaSparkContext sc)
static void	exportStatsAsHtml(SparkTrainingStats sparkTrainingStats, java.lang.String path, org.apache.spark.SparkContext sc) Generate and export a HTML representation (including charts, etc) of the Spark training statistics Note: exporting is done via Spark, so the path here can be a local file, HDFS, etc.
static java.lang.String	getDurationAsString(java.util.List<EventStats> list, java.lang.String delim)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_MAX_TIMELINE_SIZE_MS

public static final long DEFAULT_MAX_TIMELINE_SIZE_MS

See Also:

Constant Field Values

Method Detail

exportStats

public static void exportStats(java.util.List<EventStats> list,
                               java.lang.String outputDirectory,
                               java.lang.String filename,
                               java.lang.String delimiter,
                               org.apache.spark.SparkContext sc)
                        throws java.io.IOException

Throws:

java.io.IOException

exportStats

public static void exportStats(java.util.List<EventStats> list,
                               java.lang.String outputPath,
                               java.lang.String delimiter,
                               org.apache.spark.SparkContext sc)
                        throws java.io.IOException

Throws:

java.io.IOException

getDurationAsString

public static java.lang.String getDurationAsString(java.util.List<EventStats> list,
                                                   java.lang.String delim)

exportStatsAsHtml

public static void exportStatsAsHtml(SparkTrainingStats sparkTrainingStats,
                                     java.lang.String path,
                                     org.apache.spark.api.java.JavaSparkContext sc)
                              throws java.lang.Exception

Throws:

java.lang.Exception

exportStatsAsHtml

public static void exportStatsAsHtml(SparkTrainingStats sparkTrainingStats,
                                     java.lang.String path,
                                     org.apache.spark.SparkContext sc)
                              throws java.lang.Exception

Generate and export a HTML representation (including charts, etc) of the Spark training statistics
Note: exporting is done via Spark, so the path here can be a local file, HDFS, etc.

Parameters:

sparkTrainingStats - Stats to generate HTML page for

path - Path to export. May be local or HDFS

sc - Spark context

Throws:

java.lang.Exception - IO errors or error generating HTML file

exportStatsAsHtml

public static void exportStatsAsHtml(SparkTrainingStats sparkTrainingStats,
                                     long maxTimelineSizeMs,
                                     java.lang.String path,
                                     org.apache.spark.SparkContext sc)
                              throws java.lang.Exception

Generate and export a HTML representation (including charts, etc) of the Spark training statistics
Note: exporting is done via Spark, so the path here can be a local file, HDFS, etc.

Parameters:

sparkTrainingStats - Stats to generate HTML page for

path - Path to export. May be local or HDFS

maxTimelineSizeMs - maximum amount of activity to show in a single timeline plot (multiple plots will be used if training exceeds this amount of time)

sc - Spark context

Throws:

java.lang.Exception - IO errors or error generating HTML file

exportStatsAsHTML

public static void exportStatsAsHTML(SparkTrainingStats sparkTrainingStats,
                                     java.io.OutputStream outputStream)
                              throws java.lang.Exception

Generate and export a HTML representation (including charts, etc) of the Spark training statistics
This overload is for writing to an output stream

Parameters:

sparkTrainingStats - Stats to generate HTML page for

Throws:

java.lang.Exception - IO errors or error generating HTML file

exportStatsAsHTML

public static void exportStatsAsHTML(SparkTrainingStats sparkTrainingStats,
                                     long maxTimelineSizeMs,
                                     java.io.OutputStream outputStream)
                              throws java.lang.Exception

Generate and export a HTML representation (including charts, etc) of the Spark training statistics
This overload is for writing to an output stream

Parameters:

sparkTrainingStats - Stats to generate HTML page for

maxTimelineSizeMs - maximum amount of activity to show in a single timeline plot (multiple plots will be used if training exceeds this amount of time)

Throws:

java.lang.Exception - IO errors or error generating HTML file