org.datavec.spark.transform.utils

Class SparkUtils

java.lang.Object

org.datavec.spark.transform.utils.SparkUtils

public class SparkUtils
extends java.lang.Object

Created by Alex on 7/03/2016.

Constructor Summary

Constructors

Constructor and Description
SparkUtils()

Method Summary

Modifier and Type	Method and Description
static java.lang.Class<? extends org.apache.hadoop.io.compress.CompressionCodec>	getCompressionCodeClass(java.lang.String compressionCodecClass)
static <T> T	readObjectFromFile(java.lang.String path, java.lang.Class<T> type, org.apache.spark.api.java.JavaSparkContext sc) Read an object from HDFS (or local) using default Java object serialization
static <T> T	readObjectFromFile(java.lang.String path, java.lang.Class<T> type, org.apache.spark.SparkContext sc) Read an object from HDFS (or local) using default Java object serialization
static java.lang.String	readStringFromFile(java.lang.String path, org.apache.spark.api.java.JavaSparkContext sc) Read a UTF-8 format String from HDFS (or local)
static java.lang.String	readStringFromFile(java.lang.String path, org.apache.spark.SparkContext sc) Read a UTF-8 format String from HDFS (or local)
static void	registerKryoClasses(org.apache.spark.SparkConf conf) Register the DataVec writable classes for Kryo
static <T> java.util.List<org.apache.spark.api.java.JavaRDD<T>>	splitData(SplitStrategy splitStrategy, org.apache.spark.api.java.JavaRDD<T> data, long seed)
static void	writeAnalysisHTMLToFile(java.lang.String outputPath, DataAnalysis dataAnalysis, org.apache.spark.api.java.JavaSparkContext sc) Write a DataAnalysis to HDFS (or locally) as a HTML file
static void	writeObjectToFile(java.lang.String path, java.lang.Object toWrite, org.apache.spark.api.java.JavaSparkContext sc) Write an object to HDFS (or local) using default Java object serialization
static void	writeObjectToFile(java.lang.String path, java.lang.Object toWrite, org.apache.spark.SparkContext sc) Write an object to HDFS (or local) using default Java object serialization
static void	writeSchema(java.lang.String outputPath, Schema schema, org.apache.spark.api.java.JavaSparkContext sc) Write a schema to a HDFS (or, local) file in a human-readable format
static void	writeStringToFile(java.lang.String path, java.lang.String toWrite, org.apache.spark.api.java.JavaSparkContext sc) Write a String to a file (on HDFS or local) in UTF-8 format
static void	writeStringToFile(java.lang.String path, java.lang.String toWrite, org.apache.spark.SparkContext sc) Write a String to a file (on HDFS or local) in UTF-8 format
static void	writeWritablesToFile(java.lang.String outputPath, java.lang.String delim, java.util.List<java.util.List<Writable>> writables, org.apache.spark.api.java.JavaSparkContext sc) Wlite a set of writables (or, sequence) to HDFS (or, locally).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkUtils

public SparkUtils()

Method Detail

splitData

public static <T> java.util.List<org.apache.spark.api.java.JavaRDD<T>> splitData(SplitStrategy splitStrategy,
                                                                                 org.apache.spark.api.java.JavaRDD<T> data,
                                                                                 long seed)

writeStringToFile

public static void writeStringToFile(java.lang.String path,
                                     java.lang.String toWrite,
                                     org.apache.spark.api.java.JavaSparkContext sc)
                              throws java.io.IOException

Write a String to a file (on HDFS or local) in UTF-8 format

Parameters:

path - Path to write to

toWrite - String to write

sc - Spark context

Throws:

java.io.IOException

writeStringToFile

public static void writeStringToFile(java.lang.String path,
                                     java.lang.String toWrite,
                                     org.apache.spark.SparkContext sc)
                              throws java.io.IOException

Write a String to a file (on HDFS or local) in UTF-8 format

Parameters:

path - Path to write to

toWrite - String to write

sc - Spark context

Throws:

java.io.IOException

readStringFromFile

public static java.lang.String readStringFromFile(java.lang.String path,
                                                  org.apache.spark.api.java.JavaSparkContext sc)
                                           throws java.io.IOException

Read a UTF-8 format String from HDFS (or local)

Parameters:

path - Path to write the string

sc - Spark context

Throws:

java.io.IOException

readStringFromFile

public static java.lang.String readStringFromFile(java.lang.String path,
                                                  org.apache.spark.SparkContext sc)
                                           throws java.io.IOException

Read a UTF-8 format String from HDFS (or local)

Parameters:

path - Path to write the string

sc - Spark context

Throws:

java.io.IOException

writeObjectToFile

public static void writeObjectToFile(java.lang.String path,
                                     java.lang.Object toWrite,
                                     org.apache.spark.api.java.JavaSparkContext sc)
                              throws java.io.IOException

Write an object to HDFS (or local) using default Java object serialization

Parameters:

path - Path to write the object to

toWrite - Object to write

sc - Spark context

Throws:

java.io.IOException

writeObjectToFile

public static void writeObjectToFile(java.lang.String path,
                                     java.lang.Object toWrite,
                                     org.apache.spark.SparkContext sc)
                              throws java.io.IOException

Write an object to HDFS (or local) using default Java object serialization

Parameters:

path - Path to write the object to

toWrite - Object to write

sc - Spark context

Throws:

java.io.IOException

readObjectFromFile

public static <T> T readObjectFromFile(java.lang.String path,
                                       java.lang.Class<T> type,
                                       org.apache.spark.api.java.JavaSparkContext sc)
                                throws java.io.IOException

Read an object from HDFS (or local) using default Java object serialization

Type Parameters:

T - Type of the object to read

Parameters:

path - File to read

type - Class of the object to read

sc - Spark context

Throws:

java.io.IOException

readObjectFromFile

public static <T> T readObjectFromFile(java.lang.String path,
                                       java.lang.Class<T> type,
                                       org.apache.spark.SparkContext sc)
                                throws java.io.IOException

Read an object from HDFS (or local) using default Java object serialization

Type Parameters:

T - Type of the object to read

Parameters:

path - File to read

type - Class of the object to read

sc - Spark context

Throws:

java.io.IOException

writeSchema

public static void writeSchema(java.lang.String outputPath,
                               Schema schema,
                               org.apache.spark.api.java.JavaSparkContext sc)
                        throws java.io.IOException

Write a schema to a HDFS (or, local) file in a human-readable format

Parameters:

outputPath - Output path to write to

schema - Schema to write

sc - Spark context

Throws:

java.io.IOException

writeAnalysisHTMLToFile

public static void writeAnalysisHTMLToFile(java.lang.String outputPath,
                                           DataAnalysis dataAnalysis,
                                           org.apache.spark.api.java.JavaSparkContext sc)

Write a DataAnalysis to HDFS (or locally) as a HTML file

Parameters:

outputPath - Output path

dataAnalysis - Analysis to generate HTML file for

sc - Spark context

writeWritablesToFile

public static void writeWritablesToFile(java.lang.String outputPath,
                                        java.lang.String delim,
                                        java.util.List<java.util.List<Writable>> writables,
                                        org.apache.spark.api.java.JavaSparkContext sc)
                                 throws java.io.IOException

Wlite a set of writables (or, sequence) to HDFS (or, locally).

Parameters:

outputPath - Path to write the outptu

delim - Delimiter

writables - data to write

sc - Spark context

Throws:

java.io.IOException

registerKryoClasses

public static void registerKryoClasses(org.apache.spark.SparkConf conf)

Register the DataVec writable classes for Kryo

getCompressionCodeClass

public static java.lang.Class<? extends org.apache.hadoop.io.compress.CompressionCodec> getCompressionCodeClass(java.lang.String compressionCodecClass)