A 21mW low-power recurrent neural network accelerator with quantization tables for embedded deep learning applications

A 21mW low-power embedded Recurrent Neural Network (RNN) accelerator is proposed to realize the image captioning applications. The low-power RNN operation is achieved by 3 key features: 1) Quantization-table-based matrix multiplication with RNN weight quantization, 2) Dynamic quantization-table allocation scheme for balanced pipelined RNN operation, and 3) Zero-skipped RNN operation using quantization-table. The Quantization table enables the 98% reduction of the multiplier operations by replacing the multiplication to the table reference. The dynamic quantization-table allocation is used to achieve high chip-utilization efficiency over 90% by balanced pipeline operation for three variations of the RNN operation. The zero-skipped RNN operation reduces the overall 27% of required external memory bandwidth and quantization-table operations without any additional hardware cost. The proposed RNN accelerator of 1.84mm² achieves 21mW power consumption and demonstrates its functionality on the image captioning RNN in 65nm CMOS process.

I. Introduction

The deep learning is playing an important role even in the area of embedded applications such as face recognition or speech recognition. Among Deep Neural Networks (DNN s), the Recurrent Neural Network (RNN), which can process the sequential data such as speech and video, is very effective in the image captioning and visual question answering [1] together with a static object classifier such as a Convolutional Neural Network (CNN). However, in order to run deep learning applications in embedded environment of limited power and performance, a dedicated low-power, high-performance deep learning accelerator is required. Even though several dedicated accelerators for CNNs have been reported [2] [3], there is no RNN semiconductor chip because its heavy external memory bandwidth consumptions.