public class VariationalAutoencoder extends java.lang.Object implements Layer
See: Kingma & Welling, 2013: Auto-Encoding Variational Bayes - https://arxiv.org/abs/1312.6114
This implementation allows multiple encoder and decoder layers, the number and sizes of which can be set independently.
A note on scores during pretraining: This implementation minimizes the negative of the variational lower bound objective as described in Kingma & Welling; the mathematics in that paper is based on maximization of the variational lower bound instead. Thus, scores reported during pretraining in DL4J are the negative of the variational lower bound equation in the paper. The backpropagation and learning procedure is otherwise as described there.
Layer.TrainingMode, Layer.Type
Modifier and Type | Field and Description |
---|---|
protected NeuralNetConfiguration |
conf |
protected int[] |
decoderLayerSizes |
protected int[] |
encoderLayerSizes |
protected Gradient |
gradient |
protected org.nd4j.linalg.api.ndarray.INDArray |
gradientsFlattened |
protected java.util.Map<java.lang.String,org.nd4j.linalg.api.ndarray.INDArray> |
gradientViews |
protected int |
index |
protected org.nd4j.linalg.api.ndarray.INDArray |
input |
protected java.util.Collection<IterationListener> |
iterationListeners |
protected org.nd4j.linalg.api.ndarray.INDArray |
maskArray |
protected int |
numSamples |
protected ConvexOptimizer |
optimizer |
protected java.util.Map<java.lang.String,org.nd4j.linalg.api.ndarray.INDArray> |
params |
protected org.nd4j.linalg.api.ndarray.INDArray |
paramsFlattened |
protected org.nd4j.linalg.activations.IActivation |
pzxActivationFn |
protected ReconstructionDistribution |
reconstructionDistribution |
protected double |
score |
protected Solver |
solver |
protected java.util.Collection<TrainingListener> |
trainingListeners |
protected boolean |
zeroedPretrainParamGradients |
Constructor and Description |
---|
VariationalAutoencoder(NeuralNetConfiguration conf) |
Modifier and Type | Method and Description |
---|---|
void |
accumulateScore(double accum)
Sets a rolling tally for the score.
|
org.nd4j.linalg.api.ndarray.INDArray |
activate()
Trigger an activation with the last specified input
|
org.nd4j.linalg.api.ndarray.INDArray |
activate(boolean training)
Trigger an activation with the last specified input
|
org.nd4j.linalg.api.ndarray.INDArray |
activate(org.nd4j.linalg.api.ndarray.INDArray input)
Initialize the layer with the given input
and return the activation for this layer
given this input
|
org.nd4j.linalg.api.ndarray.INDArray |
activate(org.nd4j.linalg.api.ndarray.INDArray input,
boolean training)
Initialize the layer with the given input
and return the activation for this layer
given this input
|
org.nd4j.linalg.api.ndarray.INDArray |
activate(org.nd4j.linalg.api.ndarray.INDArray input,
Layer.TrainingMode training)
Initialize the layer with the given input
and return the activation for this layer
given this input
|
org.nd4j.linalg.api.ndarray.INDArray |
activate(Layer.TrainingMode training)
Trigger an activation with the last specified input
|
org.nd4j.linalg.api.ndarray.INDArray |
activationMean()
Calculate the mean representation
for the activation for this layer
|
void |
applyLearningRateScoreDecay()
Update learningRate using for this model.
|
Pair<Gradient,org.nd4j.linalg.api.ndarray.INDArray> |
backpropGradient(org.nd4j.linalg.api.ndarray.INDArray epsilon)
Calculate the gradient relative to the error in the next layer
|
int |
batchSize()
The current inputs batch size
|
Gradient |
calcGradient(Gradient layerError,
org.nd4j.linalg.api.ndarray.INDArray indArray)
Calculate the gradient
|
double |
calcL1(boolean backpropParamsOnly)
Calculate the l1 regularization term
0.0 if regularization is not used. |
double |
calcL2(boolean backpropParamsOnly)
Calculate the l2 regularization term
0.0 if regularization is not used. |
void |
clear()
Clear input
|
Layer |
clone()
Clone the layer
|
void |
computeGradientAndScore()
Update the score
|
NeuralNetConfiguration |
conf()
The configuration for the neural network
|
org.nd4j.linalg.api.ndarray.INDArray |
derivativeActivation(org.nd4j.linalg.api.ndarray.INDArray input)
Take the derivative of the given input
based on the activation
|
Gradient |
error(org.nd4j.linalg.api.ndarray.INDArray input)
Calculate error with respect to the
current layer.
|
Pair<org.nd4j.linalg.api.ndarray.INDArray,MaskState> |
feedForwardMaskArray(org.nd4j.linalg.api.ndarray.INDArray maskArray,
MaskState currentMaskState,
int minibatchSize)
Feed forward the input mask array, setting in in the layer as appropriate.
|
void |
fit()
All models have a fit method
|
void |
fit(org.nd4j.linalg.api.ndarray.INDArray data)
Fit the model to the given data
|
org.nd4j.linalg.api.ndarray.INDArray |
generateAtMeanGivenZ(org.nd4j.linalg.api.ndarray.INDArray latentSpaceValues)
Given a specified values for the latent space as input (latent space being z in p(z|data)), generate output
from P(x|z), where x = E[P(x|z)]
i.e., return the mean value for the distribution P(x|z) |
org.nd4j.linalg.api.ndarray.INDArray |
generateRandomGivenZ(org.nd4j.linalg.api.ndarray.INDArray latentSpaceValues)
Given a specified values for the latent space as input (latent space being z in p(z|data)), randomly generate output
x, where x ~ P(x|z)
|
int |
getIndex()
Get the layer index.
|
int |
getInputMiniBatchSize()
Get current/last input mini-batch size, as set by setInputMiniBatchSize(int)
|
java.util.Collection<IterationListener> |
getListeners()
Get the iteration listeners for this layer.
|
org.nd4j.linalg.api.ndarray.INDArray |
getMaskArray() |
ConvexOptimizer |
getOptimizer()
Returns this models optimizer
|
org.nd4j.linalg.api.ndarray.INDArray |
getParam(java.lang.String param)
Get the parameter
|
Gradient |
gradient()
Calculate a gradient
|
Pair<Gradient,java.lang.Double> |
gradientAndScore()
Get the gradient and score
|
boolean |
hasLossFunction()
Does the reconstruction distribution have a loss function (such as mean squared error) or is it a standard
probabilistic reconstruction distribution?
|
void |
init()
Init the model
|
void |
initParams()
Initialize the parameters
|
org.nd4j.linalg.api.ndarray.INDArray |
input()
The input/feature matrix for the model
|
boolean |
isPretrainLayer()
Returns true if the layer can be trained in an unsupervised/pretrain manner (VAE, RBMs etc)
|
boolean |
isPretrainParam(java.lang.String param) |
void |
iterate(org.nd4j.linalg.api.ndarray.INDArray input)
Run one iteration
|
void |
merge(Layer layer,
int batchSize)
Parameter averaging
|
int |
numParams()
the number of parameters for the model
|
int |
numParams(boolean backwards)
the number of parameters for the model
|
org.nd4j.linalg.api.ndarray.INDArray |
params()
Parameters of the model (if any)
|
java.util.Map<java.lang.String,org.nd4j.linalg.api.ndarray.INDArray> |
paramTable()
The param table
|
java.util.Map<java.lang.String,org.nd4j.linalg.api.ndarray.INDArray> |
paramTable(boolean backpropParamsOnly)
Table of parameters by key, for backprop
For many models (dense layers, etc) - all parameters are backprop parameters
|
org.nd4j.linalg.api.ndarray.INDArray |
preOutput(boolean training) |
org.nd4j.linalg.api.ndarray.INDArray |
preOutput(org.nd4j.linalg.api.ndarray.INDArray x)
Raw activations
|
org.nd4j.linalg.api.ndarray.INDArray |
preOutput(org.nd4j.linalg.api.ndarray.INDArray x,
boolean training)
Raw activations
|
org.nd4j.linalg.api.ndarray.INDArray |
preOutput(org.nd4j.linalg.api.ndarray.INDArray x,
Layer.TrainingMode training)
Raw activations
|
org.nd4j.linalg.api.ndarray.INDArray |
reconstructionError(org.nd4j.linalg.api.ndarray.INDArray data)
Return the reconstruction error for this variational autoencoder.
NOTE (important): This method is used ONLY for VAEs that have a standard neural network loss function (i.e., an ILossFunction instance such as mean squared error) instead of using a
probabilistic reconstruction distribution P(x|z) for the reconstructions (as presented in the VAE architecture by
Kingma and Welling).You can check if the VAE has a loss function using hasLossFunction() Consequently, the reconstruction error is a simple deterministic function (no Monte-Carlo sampling is required, unlike reconstructionProbability(INDArray, int) and reconstructionLogProbability(INDArray, int) ) |
org.nd4j.linalg.api.ndarray.INDArray |
reconstructionLogProbability(org.nd4j.linalg.api.ndarray.INDArray data,
int numSamples)
Return the log reconstruction probability given the specified number of samples.
See reconstructionLogProbability(INDArray, int) for more details |
org.nd4j.linalg.api.ndarray.INDArray |
reconstructionProbability(org.nd4j.linalg.api.ndarray.INDArray data,
int numSamples)
Calculate the reconstruction probability, as described in An & Cho, 2015 - "Variational Autoencoder based
Anomaly Detection using Reconstruction Probability" (Algorithm 4)
The authors describe it as follows: "This is essentially the probability of the data being generated from a given latent variable drawn from the approximate posterior distribution." Specifically, for each example x in the input, calculate p(x). |
double |
score()
The score for the model
|
void |
setBackpropGradientsViewArray(org.nd4j.linalg.api.ndarray.INDArray gradients)
Set the gradients array as a view of the full (backprop) network parameters
NOTE: this is intended to be used internally in MultiLayerNetwork and ComputationGraph, not by users.
|
void |
setConf(NeuralNetConfiguration conf)
Setter for the configuration
|
void |
setIndex(int index)
Set the layer index.
|
void |
setInput(org.nd4j.linalg.api.ndarray.INDArray input)
Get the layer input.
|
void |
setInputMiniBatchSize(int size)
Set current/last input mini-batch size.
Used for score and gradient calculations. |
void |
setListeners(java.util.Collection<IterationListener> listeners)
Set the iteration listeners for this layer.
|
void |
setListeners(IterationListener... listeners)
Set the iteration listeners for this layer.
|
void |
setMaskArray(org.nd4j.linalg.api.ndarray.INDArray maskArray)
Set the mask array.
|
void |
setParam(java.lang.String key,
org.nd4j.linalg.api.ndarray.INDArray val)
Set the parameter with a new ndarray
|
void |
setParams(org.nd4j.linalg.api.ndarray.INDArray params)
Set the parameters for this model.
|
void |
setParamsViewArray(org.nd4j.linalg.api.ndarray.INDArray params)
Set the initial parameters array as a view of the full (backprop) network parameters
NOTE: this is intended to be used internally in MultiLayerNetwork and ComputationGraph, not by users.
|
void |
setParamTable(java.util.Map<java.lang.String,org.nd4j.linalg.api.ndarray.INDArray> paramTable)
Setter for the param table
|
Layer |
transpose()
Return a transposed copy of the weights/bias
(this means reverse the number of inputs and outputs on the weights)
|
Layer.Type |
type()
Returns the layer type
|
void |
update(Gradient gradient)
Update layer weights and biases with gradient change
|
void |
update(org.nd4j.linalg.api.ndarray.INDArray gradient,
java.lang.String paramType)
Perform one update applying the gradient
|
void |
validateInput()
Validate the input
|
protected org.nd4j.linalg.api.ndarray.INDArray input
protected org.nd4j.linalg.api.ndarray.INDArray paramsFlattened
protected org.nd4j.linalg.api.ndarray.INDArray gradientsFlattened
protected java.util.Map<java.lang.String,org.nd4j.linalg.api.ndarray.INDArray> params
protected transient java.util.Map<java.lang.String,org.nd4j.linalg.api.ndarray.INDArray> gradientViews
protected NeuralNetConfiguration conf
protected double score
protected ConvexOptimizer optimizer
protected Gradient gradient
protected java.util.Collection<IterationListener> iterationListeners
protected java.util.Collection<TrainingListener> trainingListeners
protected int index
protected org.nd4j.linalg.api.ndarray.INDArray maskArray
protected Solver solver
protected int[] encoderLayerSizes
protected int[] decoderLayerSizes
protected ReconstructionDistribution reconstructionDistribution
protected org.nd4j.linalg.activations.IActivation pzxActivationFn
protected int numSamples
protected boolean zeroedPretrainParamGradients
public VariationalAutoencoder(NeuralNetConfiguration conf)
public void update(Gradient gradient)
Model
public void update(org.nd4j.linalg.api.ndarray.INDArray gradient, java.lang.String paramType)
Model
public double score()
Model
public void computeGradientAndScore()
Model
computeGradientAndScore
in interface Model
public void accumulateScore(double accum)
Model
accumulateScore
in interface Model
accum
- the amount to accumpublic org.nd4j.linalg.api.ndarray.INDArray params()
Model
public int numParams()
Model
public int numParams(boolean backwards)
Model
public void setParams(org.nd4j.linalg.api.ndarray.INDArray params)
Model
public void setParamsViewArray(org.nd4j.linalg.api.ndarray.INDArray params)
Model
setParamsViewArray
in interface Model
params
- a 1 x nParams row vector that is a view of the larger (MLN/CG) parameters arraypublic void setBackpropGradientsViewArray(org.nd4j.linalg.api.ndarray.INDArray gradients)
Model
setBackpropGradientsViewArray
in interface Model
gradients
- a 1 x nParams row vector that is a view of the larger (MLN/CG) gradients arraypublic void applyLearningRateScoreDecay()
Model
applyLearningRateScoreDecay
in interface Model
public void fit(org.nd4j.linalg.api.ndarray.INDArray data)
Model
public void iterate(org.nd4j.linalg.api.ndarray.INDArray input)
Model
public Pair<Gradient,java.lang.Double> gradientAndScore()
Model
gradientAndScore
in interface Model
public int batchSize()
Model
public NeuralNetConfiguration conf()
Model
public void setConf(NeuralNetConfiguration conf)
Model
public org.nd4j.linalg.api.ndarray.INDArray input()
Model
public void validateInput()
Model
validateInput
in interface Model
public ConvexOptimizer getOptimizer()
Model
getOptimizer
in interface Model
public org.nd4j.linalg.api.ndarray.INDArray getParam(java.lang.String param)
Model
public void initParams()
Model
initParams
in interface Model
public java.util.Map<java.lang.String,org.nd4j.linalg.api.ndarray.INDArray> paramTable()
Model
paramTable
in interface Model
public java.util.Map<java.lang.String,org.nd4j.linalg.api.ndarray.INDArray> paramTable(boolean backpropParamsOnly)
Model
paramTable
in interface Model
backpropParamsOnly
- If true, return backprop params only. If false: return all params (equivalent to
paramsTable())public void setParamTable(java.util.Map<java.lang.String,org.nd4j.linalg.api.ndarray.INDArray> paramTable)
Model
setParamTable
in interface Model
public void setParam(java.lang.String key, org.nd4j.linalg.api.ndarray.INDArray val)
Model
public void clear()
Model
public boolean isPretrainParam(java.lang.String param)
public double calcL2(boolean backpropParamsOnly)
Layer
public double calcL1(boolean backpropParamsOnly)
Layer
public Layer.Type type()
Layer
public Gradient error(org.nd4j.linalg.api.ndarray.INDArray input)
Layer
public org.nd4j.linalg.api.ndarray.INDArray derivativeActivation(org.nd4j.linalg.api.ndarray.INDArray input)
Layer
derivativeActivation
in interface Layer
input
- the input to take the derivative ofpublic Gradient calcGradient(Gradient layerError, org.nd4j.linalg.api.ndarray.INDArray indArray)
Layer
calcGradient
in interface Layer
layerError
- the layer errorpublic Pair<Gradient,org.nd4j.linalg.api.ndarray.INDArray> backpropGradient(org.nd4j.linalg.api.ndarray.INDArray epsilon)
Layer
backpropGradient
in interface Layer
epsilon
- w^(L+1)*delta^(L+1). Or, equiv: dC/da, i.e., (dC/dz)*(dz/da) = dC/da, where C
is cost function a=sigma(z) is activation.public void merge(Layer layer, int batchSize)
Layer
public org.nd4j.linalg.api.ndarray.INDArray activationMean()
Layer
activationMean
in interface Layer
public org.nd4j.linalg.api.ndarray.INDArray preOutput(org.nd4j.linalg.api.ndarray.INDArray x)
Layer
public org.nd4j.linalg.api.ndarray.INDArray preOutput(org.nd4j.linalg.api.ndarray.INDArray x, Layer.TrainingMode training)
Layer
public org.nd4j.linalg.api.ndarray.INDArray preOutput(org.nd4j.linalg.api.ndarray.INDArray x, boolean training)
Layer
public org.nd4j.linalg.api.ndarray.INDArray preOutput(boolean training)
public org.nd4j.linalg.api.ndarray.INDArray activate(Layer.TrainingMode training)
Layer
public org.nd4j.linalg.api.ndarray.INDArray activate(org.nd4j.linalg.api.ndarray.INDArray input, Layer.TrainingMode training)
Layer
public org.nd4j.linalg.api.ndarray.INDArray activate(boolean training)
Layer
public org.nd4j.linalg.api.ndarray.INDArray activate(org.nd4j.linalg.api.ndarray.INDArray input, boolean training)
Layer
public org.nd4j.linalg.api.ndarray.INDArray activate()
Layer
public org.nd4j.linalg.api.ndarray.INDArray activate(org.nd4j.linalg.api.ndarray.INDArray input)
Layer
public Layer transpose()
Layer
public java.util.Collection<IterationListener> getListeners()
Layer
getListeners
in interface Layer
public void setListeners(IterationListener... listeners)
Layer
setListeners
in interface Layer
setListeners
in interface Model
public void setListeners(java.util.Collection<IterationListener> listeners)
Layer
setListeners
in interface Layer
setListeners
in interface Model
public void setIndex(int index)
Layer
public int getIndex()
Layer
public void setInput(org.nd4j.linalg.api.ndarray.INDArray input)
Layer
public void setInputMiniBatchSize(int size)
Layer
setInputMiniBatchSize
in interface Layer
public int getInputMiniBatchSize()
Layer
getInputMiniBatchSize
in interface Layer
Layer.setInputMiniBatchSize(int)
public void setMaskArray(org.nd4j.linalg.api.ndarray.INDArray maskArray)
Layer
Layer.feedForwardMaskArray(INDArray, MaskState, int)
should be used in
preference to this.setMaskArray
in interface Layer
maskArray
- Mask array to setpublic org.nd4j.linalg.api.ndarray.INDArray getMaskArray()
getMaskArray
in interface Layer
public boolean isPretrainLayer()
Layer
isPretrainLayer
in interface Layer
public Pair<org.nd4j.linalg.api.ndarray.INDArray,MaskState> feedForwardMaskArray(org.nd4j.linalg.api.ndarray.INDArray maskArray, MaskState currentMaskState, int minibatchSize)
Layer
feedForwardMaskArray
in interface Layer
maskArray
- Mask array to setcurrentMaskState
- Current state of the mask - see MaskState
minibatchSize
- Current minibatch size. Needs to be known as it cannot always be inferred from the activations
array due to reshaping (such as a DenseLayer within a recurrent neural network)public void fit()
Model
public org.nd4j.linalg.api.ndarray.INDArray reconstructionProbability(org.nd4j.linalg.api.ndarray.INDArray data, int numSamples)
reconstructionLogProbability(INDArray, int)
for the actual implementation.
That method may be more numerically stable in some cases.data
- The data to calculate the reconstruction probability fornumSamples
- Number of samples with which to base the reconstruction probability on.public org.nd4j.linalg.api.ndarray.INDArray reconstructionLogProbability(org.nd4j.linalg.api.ndarray.INDArray data, int numSamples)
reconstructionLogProbability(INDArray, int)
for more detailsdata
- The data to calculate the log reconstruction probabilitynumSamples
- Number of samples with which to base the reconstruction probability on.public org.nd4j.linalg.api.ndarray.INDArray generateAtMeanGivenZ(org.nd4j.linalg.api.ndarray.INDArray latentSpaceValues)
latentSpaceValues
- Values for the latent space. size(1) must equal nOut configuration parameterpublic org.nd4j.linalg.api.ndarray.INDArray generateRandomGivenZ(org.nd4j.linalg.api.ndarray.INDArray latentSpaceValues)
latentSpaceValues
- Values for the latent space. size(1) must equal nOut configuration parameterpublic boolean hasLossFunction()
public org.nd4j.linalg.api.ndarray.INDArray reconstructionError(org.nd4j.linalg.api.ndarray.INDArray data)
ILossFunction
instance such as mean squared error) instead of using a
probabilistic reconstruction distribution P(x|z) for the reconstructions (as presented in the VAE architecture by
Kingma and Welling).hasLossFunction()
reconstructionProbability(INDArray, int)
and reconstructionLogProbability(INDArray, int)
)data
- The data to calculate the reconstruction error on