org.datavec.spark.transform

Class SparkTransformExecutor

java.lang.Object

org.datavec.spark.transform.SparkTransformExecutor

public class SparkTransformExecutor
extends java.lang.Object

Execute a datavec transform process on spark rdds.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	LOG_ERROR_PROPERTY

Constructor Summary

Constructors

Constructor and Description
SparkTransformExecutor() Deprecated. Use static methods instead of instance methods on SparkTransformExecutor

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.api.java.JavaRDD<java.util.List<Writable>>	execute(org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> inputWritables, TransformProcess transformProcess) Execute the specified TransformProcess with the given input data Note: this method can only be used if the TransformProcess returns non-sequence data.
static org.apache.spark.api.java.JavaRDD<java.util.List<Writable>>	executeJoin(Join join, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> left, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> right) Execute a join on the specified data
static org.apache.spark.api.java.JavaRDD<java.util.List<Writable>>	executeSequenceToSeparate(org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> inputSequence, TransformProcess transformProcess) Execute the specified TransformProcess with the given sequence input data Note: this method can only be used if the TransformProcess starts with sequence data, but returns non-sequential data (after reducing or converting sequential data to individual examples)
static org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>>	executeSequenceToSequence(org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> inputSequence, TransformProcess transformProcess) Execute the specified TransformProcess with the given sequence input data Note: this method can only be used if the TransformProcess starts with sequence data, and also returns sequence data
static org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>>	executeToSequence(org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> inputWritables, TransformProcess transformProcess) Execute the specified TransformProcess with the given input data Note: this method can only be used if the TransformProcess starts with non-sequential data, but returns sequence data (after grouping or converting to a sequence as one of the steps)
static boolean	isTryCatch() Returns true if the executor is in try catch mode.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG_ERROR_PROPERTY

public static final java.lang.String LOG_ERROR_PROPERTY

See Also:

Constant Field Values

Constructor Detail

SparkTransformExecutor

@Deprecated
public SparkTransformExecutor()

Deprecated. Use static methods instead of instance methods on SparkTransformExecutor

Method Detail

execute

public static org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> execute(org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> inputWritables,
                                                                                  TransformProcess transformProcess)

Execute the specified TransformProcess with the given input data
Note: this method can only be used if the TransformProcess returns non-sequence data. For TransformProcesses that return a sequence, use executeToSequence(JavaRDD, TransformProcess)

Parameters:

inputWritables - Input data to process

transformProcess - TransformProcess to execute

Returns:

Processed data

executeToSequence

public static org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> executeToSequence(org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> inputWritables,
                                                                                                            TransformProcess transformProcess)

Execute the specified TransformProcess with the given input data
Note: this method can only be used if the TransformProcess starts with non-sequential data, but returns sequence data (after grouping or converting to a sequence as one of the steps)

Parameters:

inputWritables - Input data to process

transformProcess - TransformProcess to execute

Returns:

Processed (sequence) data

executeSequenceToSeparate

public static org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> executeSequenceToSeparate(org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> inputSequence,
                                                                                                    TransformProcess transformProcess)

Execute the specified TransformProcess with the given sequence input data
Note: this method can only be used if the TransformProcess starts with sequence data, but returns non-sequential data (after reducing or converting sequential data to individual examples)

Parameters:

inputSequence - Input sequence data to process

transformProcess - TransformProcess to execute

Returns:

Processed (non-sequential) data

executeSequenceToSequence

public static org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> executeSequenceToSequence(org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> inputSequence,
                                                                                                                    TransformProcess transformProcess)

Execute the specified TransformProcess with the given sequence input data
Note: this method can only be used if the TransformProcess starts with sequence data, and also returns sequence data

Parameters:

inputSequence - Input sequence data to process

transformProcess - TransformProcess to execute

Returns:

Processed (non-sequential) data

isTryCatch

public static boolean isTryCatch()

Returns true if the executor is in try catch mode.

Returns:

executeJoin

public static org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> executeJoin(Join join,
                                                                                      org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> left,
                                                                                      org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> right)

Execute a join on the specified data

Parameters:

join - Join to execute

left - Left data for join

right - Right data for join

Returns:

Joined data