Yesterday, Deep Learning For Java (DL4J) released their new beta version, DL4J 1.0.0-beta4. The main highlight of this version is the full multi-datatype support for ND4J and DL4J, unlike past releases. The previous version deeplearning4j-1.0.0-beta3 was released last year. This 1.0.0-beta4 version also includes the addition of MKL-DNN support, new attention layers, and many more along with optimizations and bug fixes.
What’s new in DL4J 1.0.0-beta4?
Full multi-datatype support
In past releases, all N-Dimensional arrays in ND4J were limited to a single datatype, set globally. Now, arrays of all datatypes may be used simultaneously. The supported datatypes are Double, Float, Half, Long, Int, Short, Ubyte, Byte, Bool and UTF8.
CUDA Support
CUDA 10.1 support has been added and CUDA 9.0 support has been dropped. DL4J 1.0.0-beta4 also supports CUDA versions 9.2, 10.0 and 10.1. Mac (OSX) CUDA binaries are no longer provided. However, support for Linux and Windows CUDA, and OSX CPU(x86_64) is still available.
Memory Management Changes
In DL4J 1.0.0-beta4, the periodic garbage collection is disabled by default; instead, garbage collection (GC) will be called only when it is required to reclaim memory from arrays that are allocated outside of workspaces.
Deeplearning4J: Bug Fixes and Optimizations
- cuDNN helpers will no longer attempt to fall back on built-in layer implementations if an out-of-memory exception is thrown.
- Batch normalization global variance reparameterized to avoid underflow and zero/negative variance in some cases during distributed training.
- A bug where dropout instances were incorrectly shared between layers when using transfer learning with dropout has been fixed.
- An issue where tensor Along Dimension could result in an incorrect array order for edge cases and hence exceptions in LSTMs has been fixed.
ND4J and SameDiff
Features and Enhancements
- Removed reliance on periodic garbage collection calls for handling memory management of out-of-workspace (detached) INDArrays.
- New additions – TensorFlowImportValidator tool, INDArray.close() method, Nd4j.createFromNpzFile method, support for importing BERT models into SameDiff, SameDiff GraphTransformUtil, etc have been added.
- Evaluation, RegressionEvaluation etc. now support 4d (CNN segmentation) data formats.
Bug Fixes and Optimizations
- The bug with InvertMatrix.invert() with [1,1] shape matrices has been fixed.
- Edge case bug for Updater instances with length 1 state arrays has been fixed.
- In SameDiff, ‘gradients’ are now no longer defined for non-floating-point variables, and variables that aren’t required to calculate loss or parameter gradients, thus the gradient calculation performance has improved.
To know more about the release, check the detailed release notes.
Read Next
Deeplearning4j 1.0.0-alpha arrives!
7 things Java programmers need to watch for in 2019
Deep Learning Indaba presents the state of Natural Language Processing in 2018