Deeplearning4j 最新更新


Release Notes for Version 0.9.1

Deeplearning4J

  • Fixed issue with incorrect version dependencies in 0.9.0
  • Added EmnistDataSetIterator Link
  • Numerical stability improvements to LossMCXENT / LossNegativeLogLikelihood with softmax (should reduce NaNs with very large activations)

ND4J

  • Added runtime version checking for ND4J, DL4J, RL4J, Arbiter, DataVec Link

Known Issues

  • Deeplearning4j: Use of Evaluation class no-arg constructor (i.e., new Evaluation()) can result in accuracy/stats being reported as 0.0. Other Evaluation class constructors, and ComputationGraph/MultiLayerNetwork.evaluate(DataSetIterator) methods work as expected.


Release Notes for Version 0.9.0

Deeplearning4J

  • Workspaces feature added (faster training performance + less memory) Link
  • SharedTrainingMaster added for Spark network training (improved performance) Link 1, Link 2
  • ParallelInference added - wrapper that server inference requests using internal batching and queues Link
  • ParallelWrapper now able to work with gradients sharing, in addition to existing parameters averaging mode Link
  • VPTree performance significantly improved
  • CacheMode network configuration option added - improved CNN and LSTM performance at the expense of additional memory use Link
  • LSTM layer added, with CuDNN support Link (Note that the existing GravesLSTM implementation does not support CuDNN)
  • New native model zoo with pretrained ImageNet, MNIST, and VGG-Face weights Link
  • Convolution performance improvements, including activation caching
  • Custom/user defined updaters are now supported Link
  • Evaluation improvements
    • EvaluationBinary, ROCBinary classes added: for evaluation of binary multi-class networks (sigmoid + xent output layers) Link
    • Evaluation and others now have G-Measure and Matthews Correlation Coefficient support; also macro + micro-averaging support for Evaluation class metrics Link
    • ComputationGraph and SparkComputationGraph evaluation convenience methods added (evaluateROC, etc)
    • ROC and ROCMultiClass support exact calculation (previous: thresholded calculation was used) Link
    • ROC classes now support area under precision-recall curve calculation; getting precision/recall/confusion matrix at specified thresholds (via PrecisionRecallCurve class) Link
    • RegressionEvaluation, ROCBinary etc now support per-output masking (in addition to per-example/per-time-step masking)
    • EvaluationCalibration added (residual plots, reliability diagrams, histogram of probabilities) Link 1 Link 2
    • Evaluation and EvaluationBinary: now supports custom classification threshold or cost array Link
  • Optimizations: updaters, bias calculation
  • Network memory estimation functionality added. Memory requirements can be estimated from configuration without instantiating networks Link 1 Link 2
  • New loss functions:
    • Mixture density loss function Link
    • F-Measure loss function Link

ND4J

  • Workspaces feature added Link
  • Native parallel sort was added
  • New ops added: SELU/SELUDerivative, TAD-based comparisons, percentile/median, Reverse, Tan/TanDerivative, SinH, CosH, Entropy, ShannonEntropy, LogEntropy, AbsoluteMin/AbsoluteMax/AbsoluteSum, Atan2
  • New distance functions added: CosineDistance, HammingDistance, JaccardDistance

DataVec

  • MapFileRecordReader and MapFileSequenceRecordReader added Link 1 Link 2
  • Spark: Utilities to save and load JavaRDD<List<Writable>> and JavaRDD<List<List<Writable>> data to Hadoop MapFile and SequenceFile formats Link
  • TransformProcess and Transforms now support NDArrayWritables and NDArrayWritable columns
  • Multiple new Transform classes

Arbiter

  • Arbiter UI: Link
    • UI now uses Play framework, integrates with DL4J UI (replaces Dropwizard backend). Dependency issues/clashing versions fixed.
    • Supports DL4J StatsStorage and StatsStorageRouter mechanisms (FileStatsStorage, Remote UI via RemoveUIStatsStorageRouter)
    • General UI improvements (additional information, formatting fixes)


Release Notes for Version 0.8.0

  • Added transfer learning API Link
  • Spark 2.0 support (DL4J and DataVec; see transition notes below)
  • New layers
    • Global pooling (aka “pooling over time”; usable with both RNNs and CNNs) Link
    • Center loss output layer Link
    • 1D Convolution and subsampling layers Link Link2
    • ZeroPaddingLayer Link
  • New ComputationGraph vertices
    • L2 distance vertex
    • L2 normalization vertex
  • Per-output masking is now supported for most loss functions (for per output masking, use a mask array equal in size/shape to the labels array; previous masking functionality was per-example for RNNs)
  • L1 and L2 regularization can now be configured for biases (via l1Bias and l2Bias configuration options)
  • Evaluation improvements:
    • DL4J now has an IEvaluation class (that Evaluation, RegressionEvaluation, etc all implement. Also allows custom evaluation on Spark) Link
    • Added multi-class (one vs. all) ROC: ROCMultiClass Link
    • For both MultiLayerNetwork and SparkDl4jMultiLayer: added evaluateRegression, evaluateROC, evaluateROCMultiClass convenience methods
    • HTML export functionality added for ROC charts Link
    • TSNE re-added to new UI
    • Training UI: now usable without an internet connection (no longer relies on externally hosted fonts)
    • UI: improvements to error handling for ‘no data’ condition
  • Epsilon configuration now used for Adam and RMSProp updaters
  • Fix for bidirectional LSTMs + variable-length time series (using masking)
  • Added CnnSentenceDataSetIterator (for use with ‘CNN for Sentence Classification’ architecture) Link Link2
  • Spark + Kryo: now test serialization + throw exception if misconfigured (instead of logging an error that can be missed)
  • MultiLayerNetwork now adds default layer names if no name is specified
  • DataVec:
    • JSON/YAML support for DataAnalysis, custom Transforms etc
    • ImageRecordReader refactored to reduce garbage collection load (hence improve performance with large training sets)
    • Faster quality analysis.
  • Arbiter: added new layer types to match DL4J
    • Performance improvement for Word2Vec/ParagraphVectors tokenization & training.
  • Batched inference introduced for ParagraphVectors
  • Nd4j improvements
    • New native operations available for ND4j: firstIndex, lastIndex, remainder, fmod, or, and, xor.
    • OpProfiler NAN_PANIC & INF_PANIC now also checks result of BLAS calls.
    • Nd4.getMemoryManager() now provides methods to tweak GC behavior.
  • Alpha version of parameter server for Word2Vec/ParagraphVectors were introduced for Spark. Please note: It’s not recommended for production use yet.
  • Performance improvements for CNN inference


0.7.2 -> 0.8.0 Transition Notes

Spark versioning schemes: with the addition of Spark 2 support, the versions for Deeplearning4j and DataVec Spark modules has changed

  • For Spark 1: use <version>0.8.0_spark_1</version>
  • For Spark 2: use <version>0.8.0_spark_2</version>
  • Also note: Modules with Spark 2 support are released with Scala 2.11 support only. Spark 1 modules are released with both Scala 2.10 and 2.11 support


0.8.0 Known Issues (At Launch)

  • UI/CUDA/Linux issue: Link
  • Dirty shutdown on JVM exit is possible for CUDA backend sometimes: Link
  • Issues with RBM implementation Link


Release Notes for Version 0.7.2

  • Added variational autoencoder Link
  • Activation function refactor
    • Activation functions are now an interface Link
    • Configuration now via enumeration, not via String (see examples - Link)
    • Custom activation functions now supported Link
    • New activation functions added: hard sigmoid, randomized leaky rectified linear units (RReLU)
  • Multiple fixes/improvements for Keras model import
  • Added P-norm pooling for CNNs (option as part of SubsamplingLayer configuration)
  • Iteration count persistence: stored/persisted properly in model configuration + fixes to learning rate schedules for Spark network training
  • LSTM: gate activation function can now be configured (previously: hard-coded to sigmoid)
  • UI:
    • Added Chinese translation
    • Fixes for UI + pretrain layers
    • Added Java 7 compatible stats collection compatibility Link
    • Improvements in front-end for handling NaNs
    • Added UIServer.stop() method
    • Fixed score vs. iteration moving average line (with subsampling)
  • Solved Jaxb/Jackson issue with Spring Boot based applications
  • RecordReaderDataSetIterator now supports NDArrayWritable for the labels (set regression == true; used for multi-label classification + images, etc)

0.7.1 -> 0.7.2 Transition Notes

Activation functions (built-in): now specified using Activation enumeration, not String (String-based configuration has been deprecated)


Release Notes for Version 0.7.1

  • RBM and AutoEncoder key fixes:
    • Ensured visual bias updated and applied during pretraining.
    • RBM HiddenUnit is the activation function for this layer; thus, established derivative calculations for backprop according to respective HiddenUnit.
  • RNG performance issues fixed for CUDA backend
  • OpenBLAS issues fixed for macOS, powerpc, linux.
  • DataVec is back to Java 7 now.
  • Multiple minor bugs fixed for ND4J/DL4J

Release Notes for Version 0.7.0

  • UI overhaul: new training UI has considerably more information, supports persistence (saving info and loading later), Japanese/Korean/Russian support. Replaced Dropwizard with Play framework. Link
  • Import of models configured and trained using Keras
  • Added ‘Same’ padding more for CNNs (ConvolutionMode network configuration option) Link
  • Weighted loss functions: Loss functions now support a per-output weight array (row vector)
  • ROC and AUC added for binary classifiers Link
  • Improved error messages on invalid configuration or data; improved validation on both
  • Added metadata functionality: track source of data (file, line number, etc) from data import to evaluation. Loading a subset of examples/data from this metadata is now supported. Link
  • Removed Jackson as core dependency (shaded); users can now use any version of Jackson without issue
  • Added LossLayer: version of OutputLayer that only applies loss function (unlike OutputLayer: it has no weights/biases)
  • Functionality required to build triplet embedding model (L2 vertex, LossLayer, Stack/Unstack vertices etc)
  • Reduced DL4J and ND4J ‘cold start’ initialization/start-up time
  • Pretrain default changed to false and backprop default changed to true. No longer needed to set these when setting up a network configuration unless defaults need to be changed.
  • Added TrainingListener interface (extends IterationListener). Provides access to more information/state as network training occurs Link
  • Numerous bug fixes across DL4J and ND4J
  • Performance improvements for nd4j-native & nd4j-cuda backends
  • Standalone Word2Vec/ParagraphVectors overhaul:
    • Performance improvements
    • ParaVec inference available for both PV-DM & PV-DBOW
    • Parallel tokenization support was added, to address computation-heavy tokenizers.
  • Native RNG introduced for better reproducibility within multi-threaded execution environment.
  • Additional RNG calls added: Nd4j.choice(), and BernoulliDistribution op.
  • Off-gpu storage introduced, to keep large things, like Word2Vec model in host memory. Available via WordVectorSerializer.loadStaticModel()
  • Two new options for performance tuning on nd4j-native backend: setTADThreshold(int) & setElementThreshold(int)

0.6.0 -> 0.7.0 Transition Notes

Notable changes for upgrading codebases based on 0.6.0 to 0.7.0:

  • UI: new UI package name is deeplearning4j-ui_2.10 or deeplearning4j-ui_2.11 (previously: deeplearning4j-ui). Scala version suffix is necessary due to Play framework (written in Scala) being used now.
  • Histogram and Flow iteration listeners deprecated. They are still functional, but using new UI is recommended Link
  • DataVec ImageRecordReader: labels are now sorted alphabetically by default before assigning an integer class index to each - previously (0.6.0 and earlier) they were according to file iteration order. Use .setLabels(List) to manually specify the order if required.
  • CNNs: configuration validation is now less strict. With new ConvolutionMode option, 0.6.0 was equivalent to ‘Strict’ mode, but new default is ‘Truncate’
    • See ConvolutionMode javadoc for more details: Link
  • Xavier weight initialization change for CNNs and LSTMs: Xavier now aligns better with original Glorot paper and other libraries. Xavier weight init. equivalent to 0.6.0 is available as XAVIER_LEGACY
  • DataVec: Custom RecordReader and SequenceRecordReader classes require additional methods, for the new metadata functionality. Refer to existing record reader implementations for how to implement these methods.
  • Word2Vec/ParagraphVectors:
    • Few new builder methods:
      • allowParallelTokenization(boolean)
      • useHierarchicSoftmax(boolean)
    • Behaviour change: batchSize: now batch size is ALSO used as threshold to execute number of computational batches for sg/cbow

Release Notes for Version 0.6.0

  • Custom layer support
  • Support for custom loss functions
  • Support for compressed INDArrays, for memory saving on huge data
  • Native support for BooleanIndexing where applicable
  • Initial support for combined operations on CUDA
  • Significant performance improvements on CPU & CUDA backends
  • Better support for Spark environments using CUDA & cuDNN with multi-gpu clusters
  • New UI tools: FlowIterationListener and ConvolutionIterationListener, for better insights of processes within NN.
  • Special IterationListener implementation for performance tracking: PerformanceListener
  • Inference implementation added for ParagraphVectors, together with option to use existing Word2Vec model
  • Severely decreased file size on the deeplearnning4j api
  • nd4j-cuda-8.0 backend is available now for cuda 8 RC
  • Added multiple new built-in loss functions
  • Custom preprocessor support
  • Performance improvements to Spark training implementation
  • Improved network configuration validation using InputType functionality

Release Notes for Version 0.5.0

  • FP16 support for CUDA
  • [Better performance for multi-gpu}(http://deeplearning4j.org/gpu)
  • Including optional P2P memory access support
  • Normalization support for time series and images
  • Normalization support for labels
  • Removal of Canova and shift to DataVec: Javadoc, Github Repo
  • Numerous bug fixes
  • Spark improvements

Release Notes for version 0.4.0

  • Initial multi-GPU support viable for standalone and Spark.
  • Refactored the Spark API significantly
  • Added CuDNN wrapper
  • Performance improvements for ND4J
  • Introducing DataVec: Lots of new functionality for transforming, preprocessing, cleaning data. (This replaces Canova)
  • New DataSetIterators for feeding neural nets with existing data: ExistingDataSetIterator, Floats(Double)DataSetIterator, IteratorDataSetIterator
  • New learning algorithms for word2vec and paravec: CBOW and PV-DM respectively
  • New native ops for better performance: DropOut, DropOutInverted, CompareAndSet, ReplaceNaNs
  • Shadow asynchronous datasets prefetch enabled by default for both MultiLayerNetwork and ComputationGraph
  • Better memory handling with JVM GC and CUDA backend, resulting in significantly lower memory footprint

资源


Roadmap for Fall 2016

  • ScalNet Scala API (WIP!)
  • Standard NN configuration file shared with Keras
  • CGANs
  • Model interpretability

 

与我们在Gitter聊天