Hybrid In-Memory Computing Architecture for the Training of Deep Neural Networks

The cost involved in training deep neural networks (DNNs) on von-Neumann architectures has motivated the development of novel solutions for efficient DNN training accelerators. We propose a hybrid in-memory computing (HIC) architecture for the training of DNNs on hardware accelerators that results in memory-efficient inference and outperforms baseline software accuracy in benchmark tasks. We introduce a weight representation technique that exploits both binary and multi-level phase-change memory (PCM) devices, and this leads to a memory-efficient inference accelerator. Unlike previous in-memory computing- based implementations, we use a low precision weight update accumulator that results in more memory savings. We trained the ResNet-32 network to classify CIFAR-10 images using HIC. For a comparable model size, HIC-based training outperforms baseline network, trained in floating-point 32-bit (FP32) precision, by leveraging appropriate network width multiplier. Furthermore, we observe that HIC-based training results in about 50 % less inference model size to achieve baseline comparable accuracy. We also show that the temporal drift in PCM devices has a negligible effect on post-training inference accuracy for extended periods (year). Finally, our simulations indicate HIC-based training naturally ensures that the number of write-erase cycles seen by the devices is a small fraction of the endurance limit of PCM, demonstrating the feasibility of this architecture for achieving hardware platforms that can learn in the field.

I. Introduction

Numerous emerging smart applications (e.g. IoT, wearables, drones, etc.) demand on-chip continuous learning, compelling the development of application-specific memories and architectures. More often these applications demand the implementation of learning algorithms for large network models in an energy-efficient manner. Conventional digital memory solutions based on SRAM or DRAM cannot address the required density/energy requirements due to large area and restrictive off-chip memory access costs. High-end expensive graphical processing units (GPUs) have been a default choice to perform DNN training. The energy and time requirement of training the state-of-the-art DNN architectures on GPUs is high [1]. This necessitates the development of more energy-/area-efficient custom hardware accelerators for performing deep learning training workloads. A few ASIC processors have been recently reported for DNN training [2], [3], [4], but based on conventional SRAM for on-chip storage, which requires a large amount of memory access with associated density and leakage power constraints.