org.datavec.spark.transform

Class AnalyzeSpark

java.lang.Object

org.datavec.spark.transform.AnalyzeSpark

public class AnalyzeSpark
extends java.lang.Object

AnalizeSpark: static methods for analyzing and processing RDD<List<Writable>> and RDD<List<List<Writable>>

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_HISTOGRAM_BUCKETS

Constructor Summary

Constructors

Constructor and Description
AnalyzeSpark()

Method Summary

Modifier and Type	Method and Description
static DataAnalysis	analyze(Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data)
static DataAnalysis	analyze(Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data, int maxHistogramBuckets)
static DataQualityAnalysis	analyzeQuality(Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data)
static DataQualityAnalysis	analyzeQualitySequence(Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> data)
static SequenceDataAnalysis	analyzeSequence(Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> data)
static SequenceDataAnalysis	analyzeSequence(Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> data, int maxHistogramBuckets)
static java.util.List<Writable>	getUnique(java.lang.String columnName, Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data) Get a list of unique values from the specified column.
static java.util.List<Writable>	getUniqueSequence(java.lang.String columnName, Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> sequenceData) Get a list of unique values from the specified column of a sequence
static java.util.List<java.util.List<Writable>>	sample(int count, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data) Randomly sample a set of examples
static java.util.List<Writable>	sampleFromColumn(int count, java.lang.String columnName, Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data) Randomly sample values from a single column
static java.util.List<Writable>	sampleFromColumnSequence(int count, java.lang.String columnName, Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> sequenceData) Randomly sample values from a single column, in all sequences.
static java.util.List<Writable>	sampleInvalidFromColumn(int numToSample, java.lang.String columnName, Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data) Randomly sample a set of invalid values from a specified column.
static java.util.List<Writable>	sampleInvalidFromColumn(int numToSample, java.lang.String columnName, Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data, boolean ignoreMissing) Randomly sample a set of invalid values from a specified column.
static java.util.List<Writable>	sampleInvalidFromColumnSequence(int numToSample, java.lang.String columnName, Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> data) Randomly sample a set of invalid values from a specified column, for a sequence data set.
static java.util.Map<Writable,java.lang.Long>	sampleMostFrequentFromColumn(int nMostFrequent, java.lang.String columnName, Schema schema, org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data) Sample the N most frequently occurring values in the specified column
static java.util.List<java.util.List<java.util.List<Writable>>>	sampleSequence(int count, org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> data) Randomly sample a number of sequences from the data

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_HISTOGRAM_BUCKETS

public static final int DEFAULT_HISTOGRAM_BUCKETS

See Also:

Constant Field Values

Constructor Detail

AnalyzeSpark

public AnalyzeSpark()

Method Detail

analyzeSequence

public static SequenceDataAnalysis analyzeSequence(Schema schema,
                                                   org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> data)

analyzeSequence

public static SequenceDataAnalysis analyzeSequence(Schema schema,
                                                   org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> data,
                                                   int maxHistogramBuckets)

Parameters:

schema -

data -

maxHistogramBuckets -

Returns:

analyze

public static DataAnalysis analyze(Schema schema,
                                   org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data)

analyze

public static DataAnalysis analyze(Schema schema,
                                   org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data,
                                   int maxHistogramBuckets)

sampleFromColumn

public static java.util.List<Writable> sampleFromColumn(int count,
                                                        java.lang.String columnName,
                                                        Schema schema,
                                                        org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data)

Randomly sample values from a single column

Parameters:

count - Number of values to sample

columnName - Name of the column to sample from

schema - Schema

data - Data to sample from

Returns:

A list of random samples

sampleFromColumnSequence

public static java.util.List<Writable> sampleFromColumnSequence(int count,
                                                                java.lang.String columnName,
                                                                Schema schema,
                                                                org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> sequenceData)

Randomly sample values from a single column, in all sequences. Values may be taken from any sequence (i.e., sequence order is not preserved)

Parameters:

count - Number of values to sample

columnName - Name of the column to sample from

schema - Schema

sequenceData - Data to sample from

Returns:

A list of random samples

getUnique

public static java.util.List<Writable> getUnique(java.lang.String columnName,
                                                 Schema schema,
                                                 org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data)

Get a list of unique values from the specified column. For sequence data, use getUniqueSequence(String, Schema, JavaRDD)

Parameters:

columnName - Name of the column to get unique values from

schema - Data schema

data - Data to get unique values from

Returns:

List of unique values

getUniqueSequence

public static java.util.List<Writable> getUniqueSequence(java.lang.String columnName,
                                                         Schema schema,
                                                         org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> sequenceData)

Get a list of unique values from the specified column of a sequence

Parameters:

columnName - Name of the column to get unique values from

schema - Data schema

sequenceData - Sequence data to get unique values from

Returns:

sample

public static java.util.List<java.util.List<Writable>> sample(int count,
                                                              org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data)

Randomly sample a set of examples

Parameters:

count - Number of samples to generate

data - Data to sample from

Returns:

Samples

sampleSequence

public static java.util.List<java.util.List<java.util.List<Writable>>> sampleSequence(int count,
                                                                                      org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> data)

Randomly sample a number of sequences from the data

Parameters:

count - Number of sequences to sample

data - Data to sample from

Returns:

Sequence samples

analyzeQualitySequence

public static DataQualityAnalysis analyzeQualitySequence(Schema schema,
                                                         org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> data)

Parameters:

schema -

data -

Returns:

analyzeQuality

public static DataQualityAnalysis analyzeQuality(Schema schema,
                                                 org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data)

Parameters:

schema -

data -

Returns:

sampleInvalidFromColumn

public static java.util.List<Writable> sampleInvalidFromColumn(int numToSample,
                                                               java.lang.String columnName,
                                                               Schema schema,
                                                               org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data)

Randomly sample a set of invalid values from a specified column. Values are considered invalid according to the Schema / ColumnMetaData

Parameters:

numToSample - Maximum number of invalid values to sample

columnName - Same of the column from which to sample invalid values

schema - Data schema

data - Data

Returns:

List of invalid examples

sampleInvalidFromColumn

public static java.util.List<Writable> sampleInvalidFromColumn(int numToSample,
                                                               java.lang.String columnName,
                                                               Schema schema,
                                                               org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data,
                                                               boolean ignoreMissing)

Randomly sample a set of invalid values from a specified column. Values are considered invalid according to the Schema / ColumnMetaData

Parameters:

numToSample - Maximum number of invalid values to sample

columnName - Same of the column from which to sample invalid values

schema - Data schema

data - Data

ignoreMissing - If true: ignore missing values (NullWritable or empty/null string) when sampling. If false: include missing values in sampling

Returns:

List of invalid examples

sampleInvalidFromColumnSequence

public static java.util.List<Writable> sampleInvalidFromColumnSequence(int numToSample,
                                                                       java.lang.String columnName,
                                                                       Schema schema,
                                                                       org.apache.spark.api.java.JavaRDD<java.util.List<java.util.List<Writable>>> data)

Randomly sample a set of invalid values from a specified column, for a sequence data set. Values are considered invalid according to the Schema / ColumnMetaData

Parameters:

numToSample - Maximum number of invalid values to sample

columnName - Same of the column from which to sample invalid values

schema - Data schema

data - Data

Returns:

List of invalid examples

sampleMostFrequentFromColumn

public static java.util.Map<Writable,java.lang.Long> sampleMostFrequentFromColumn(int nMostFrequent,
                                                                                  java.lang.String columnName,
                                                                                  Schema schema,
                                                                                  org.apache.spark.api.java.JavaRDD<java.util.List<Writable>> data)

Sample the N most frequently occurring values in the specified column

Parameters:

nMostFrequent - Top N values to sample

columnName - Name of the column to sample from

schema - Schema of the data

data - RDD containing the data

Returns:

List of the most frequently occurring Writable objects in that column, along with their counts