org.datavec.image.loader

Class CifarLoader

java.lang.Object

org.datavec.image.loader.BaseImageLoader

org.datavec.image.loader.NativeImageLoader

org.datavec.image.loader.CifarLoader

All Implemented Interfaces:

java.io.Serializable

public class CifarLoader
extends NativeImageLoader
implements java.io.Serializable

CifarLoader is loader specific for the Cifar10 dataset Reference: Learning Multiple Layers of Features from Tiny Images, Alex Krizhevsky, 2009. There is a special preProcessor used to normalize the dataset based on Sergey Zagoruyko example https://github.com/szagoruyko/cifar.torch

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	BYTEFILELEN
protected static int	channels
static int	CHANNELS
static java.util.Map<java.lang.String,java.lang.String>	cifarDataMap
static java.lang.String	dataBinFile
static java.lang.String	dataBinUrl
protected int	fileNum
static java.io.File	fullDir
protected static int	height
static int	HEIGHT
protected static java.util.List<org.nd4j.linalg.dataset.DataSet>	inputBatched
protected static java.io.InputStream	inputStream
protected static java.lang.String	labelFileName
protected static java.util.List<java.lang.String>	labels
protected org.nd4j.linalg.dataset.DataSet	loadDS
protected int	loadDSIndex
static java.lang.String	localDir
protected boolean	meanStdStored
static java.io.File	meanVarPath
static int	NUM_LABELS
static int	NUM_TEST_IMAGES
static int	NUM_TRAIN_IMAGES
protected int	numExamples
protected static int	numToConvertDS
protected static long	seed
protected static boolean	shuffle
static java.lang.String	TESTFILENAME
protected static java.lang.String	testFilesSerialized
protected static java.io.InputStream	testInputStream
protected static boolean	train
static java.lang.String[]	TRAINFILENAMES
protected static java.lang.String	trainFilesSerialized
protected static java.io.InputStream	trainInputStream
protected double	uMean
static boolean	useSpecialPreProcessCifar
protected double	uStd
protected double	vMean
protected double	vStd
protected static int	width
static int	WIDTH

Fields inherited from class org.datavec.image.loader.NativeImageLoader

ALLOWED_FORMATS

Fields inherited from class org.datavec.image.loader.BaseImageLoader

BASE_DIR, centerCropIfNeeded, imageTransform, log, rng

Constructor Summary

Constructors

Constructor and Description
CifarLoader()
CifarLoader(boolean train)
CifarLoader(boolean train, java.io.File fullPath)
CifarLoader(int height, int width, int channels, boolean train, boolean useSpecialPreProcessCifar)
CifarLoader(int height, int width, int channels, ImageTransform imgTransform, boolean train, boolean useSpecialPreProcessCifar)
CifarLoader(int height, int width, int channels, ImageTransform imgTransform, boolean train, boolean useSpecialPreProcessCifar, boolean shuffle)
CifarLoader(int height, int width, int channels, ImageTransform imgTransform, boolean train, boolean useSpecialPreProcessCifar, java.io.File fullPath, long seed, boolean shuffle)

Method Summary

Modifier and Type	Method and Description
org.nd4j.linalg.api.ndarray.INDArray	asMatrix(java.io.File f)
org.nd4j.linalg.api.ndarray.INDArray	asMatrix(java.io.InputStream inputStream)
org.nd4j.linalg.api.ndarray.INDArray	asRowVector(java.io.File f) Convert a file to a row vector
org.nd4j.linalg.api.ndarray.INDArray	asRowVector(java.io.InputStream inputStream)
boolean	cifarRawFilesExist()
org.bytedeco.javacpp.opencv_core.Mat	convertCifar(org.bytedeco.javacpp.opencv_core.Mat orgImage) Preprocess and store cifar based on successful Torch approach by Sergey Zagoruyko Reference: https://github.com/szagoruyko/cifar.torch
org.nd4j.linalg.dataset.DataSet	convertDataSet(int num)
Pair<org.nd4j.linalg.api.ndarray.INDArray,org.bytedeco.javacpp.opencv_core.Mat>	convertMat(byte[] byteFeature)
void	generateMaps()
java.io.InputStream	getInputStream()
java.util.List<java.lang.String>	getLabels()
void	load()
org.nd4j.linalg.dataset.DataSet	next(int batchSize)
org.nd4j.linalg.dataset.DataSet	next(int batchSize, int exampleNum)
void	normalizeCifar(java.io.File fileName) Normalize and store cifar based on successful Torch approach by Sergey Zagoruyko Reference: https://github.com/szagoruyko/cifar.torch
void	reset()
void	setInputStream()
void	test()
void	train()
double	varManual(org.nd4j.linalg.api.ndarray.INDArray x, double mean)

Methods inherited from class org.datavec.image.loader.NativeImageLoader

asMatrix, asMatrix, asRowVector, asRowVector, centerCropIfNeeded, getAllowedFormats, scalingIfNeed, scalingIfNeed

Methods inherited from class org.datavec.image.loader.BaseImageLoader

downloadAndUntar

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NUM_TRAIN_IMAGES

public static final int NUM_TRAIN_IMAGES

See Also:

Constant Field Values

NUM_TEST_IMAGES

public static final int NUM_TEST_IMAGES

See Also:

Constant Field Values

NUM_LABELS

public static final int NUM_LABELS

See Also:

Constant Field Values

HEIGHT

public static final int HEIGHT

See Also:

Constant Field Values

WIDTH

public static final int WIDTH

See Also:

Constant Field Values

CHANNELS

public static final int CHANNELS

See Also:

Constant Field Values

BYTEFILELEN

public static final int BYTEFILELEN

See Also:

Constant Field Values

dataBinUrl

public static java.lang.String dataBinUrl

localDir

public static java.lang.String localDir

dataBinFile

public static java.lang.String dataBinFile

fullDir

public static java.io.File fullDir

meanVarPath

public static java.io.File meanVarPath

labelFileName

protected static java.lang.String labelFileName

inputStream

protected static java.io.InputStream inputStream

trainInputStream

protected static java.io.InputStream trainInputStream

testInputStream

protected static java.io.InputStream testInputStream

inputBatched

protected static java.util.List<org.nd4j.linalg.dataset.DataSet> inputBatched

labels

protected static java.util.List<java.lang.String> labels

TRAINFILENAMES

public static java.lang.String[] TRAINFILENAMES

TESTFILENAME

public static java.lang.String TESTFILENAME

trainFilesSerialized

protected static java.lang.String trainFilesSerialized

testFilesSerialized

protected static java.lang.String testFilesSerialized

train

protected static boolean train

useSpecialPreProcessCifar

public static boolean useSpecialPreProcessCifar

cifarDataMap

public static java.util.Map<java.lang.String,java.lang.String> cifarDataMap

height

protected static int height

width

protected static int width

channels

protected static int channels

seed

protected static long seed

shuffle

protected static boolean shuffle

numExamples

protected int numExamples

numToConvertDS

protected static int numToConvertDS

uMean

protected double uMean

uStd

protected double uStd

vMean

protected double vMean

vStd

protected double vStd

meanStdStored

protected boolean meanStdStored

loadDSIndex

protected int loadDSIndex

loadDS

protected org.nd4j.linalg.dataset.DataSet loadDS

fileNum

protected int fileNum

Constructor Detail

CifarLoader

public CifarLoader()

CifarLoader

public CifarLoader(boolean train)

CifarLoader

public CifarLoader(boolean train,
                   java.io.File fullPath)

CifarLoader

public CifarLoader(int height,
                   int width,
                   int channels,
                   boolean train,
                   boolean useSpecialPreProcessCifar)

CifarLoader

public CifarLoader(int height,
                   int width,
                   int channels,
                   ImageTransform imgTransform,
                   boolean train,
                   boolean useSpecialPreProcessCifar)

CifarLoader

public CifarLoader(int height,
                   int width,
                   int channels,
                   ImageTransform imgTransform,
                   boolean train,
                   boolean useSpecialPreProcessCifar,
                   boolean shuffle)

CifarLoader

public CifarLoader(int height,
                   int width,
                   int channels,
                   ImageTransform imgTransform,
                   boolean train,
                   boolean useSpecialPreProcessCifar,
                   java.io.File fullPath,
                   long seed,
                   boolean shuffle)

Method Detail

asRowVector

public org.nd4j.linalg.api.ndarray.INDArray asRowVector(java.io.File f)
                                                 throws java.io.IOException

Description copied from class: NativeImageLoader

Convert a file to a row vector

Overrides:

asRowVector in class NativeImageLoader

Parameters:

f - the image to convert

Returns:

the flattened image

Throws:

java.io.IOException

asRowVector

public org.nd4j.linalg.api.ndarray.INDArray asRowVector(java.io.InputStream inputStream)
                                                 throws java.io.IOException

Overrides:

asRowVector in class NativeImageLoader

Throws:

java.io.IOException

asMatrix

public org.nd4j.linalg.api.ndarray.INDArray asMatrix(java.io.File f)
                                              throws java.io.IOException

Overrides:

asMatrix in class NativeImageLoader

Throws:

java.io.IOException

asMatrix

public org.nd4j.linalg.api.ndarray.INDArray asMatrix(java.io.InputStream inputStream)
                                              throws java.io.IOException

Overrides:

asMatrix in class NativeImageLoader

Throws:

java.io.IOException

generateMaps

public void generateMaps()

load

public void load()

cifarRawFilesExist

public boolean cifarRawFilesExist()

convertCifar

public org.bytedeco.javacpp.opencv_core.Mat convertCifar(org.bytedeco.javacpp.opencv_core.Mat orgImage)

Preprocess and store cifar based on successful Torch approach by Sergey Zagoruyko Reference: https://github.com/szagoruyko/cifar.torch

normalizeCifar

public void normalizeCifar(java.io.File fileName)

Normalize and store cifar based on successful Torch approach by Sergey Zagoruyko Reference: https://github.com/szagoruyko/cifar.torch

convertMat

public Pair<org.nd4j.linalg.api.ndarray.INDArray,org.bytedeco.javacpp.opencv_core.Mat> convertMat(byte[] byteFeature)

convertDataSet

public org.nd4j.linalg.dataset.DataSet convertDataSet(int num)

varManual

public double varManual(org.nd4j.linalg.api.ndarray.INDArray x,
                        double mean)

next

public org.nd4j.linalg.dataset.DataSet next(int batchSize)

next

public org.nd4j.linalg.dataset.DataSet next(int batchSize,
                                            int exampleNum)

getInputStream

public java.io.InputStream getInputStream()

setInputStream

public void setInputStream()

getLabels

public java.util.List<java.lang.String> getLabels()

reset

public void reset()

train

public void train()

test

public void test()