REX-SC: Range-Extended Stochastic Computing Accumulation for Neural Network Acceleration

Deep learning has grown in capability and size in recent years, prompting research on alternative computing methods to cope with the increased compute cost. Stochastic computing (SC) promises higher compute efficiency with its compact compute units, but accuracy issues have prevented wide adoption, and accuracy-improving techniques have sacrificed runtime or training performance. In this work, we propose extended range SC—Range-Extended SC Accumulation to deal with the accuracy issues of SC. By modifying the functionality of OR-based SC accumulation, we increase SC computation accuracy without sacrificing the performance benefits. Our approach achieves a $2\times $ reduction in stream length for the same accuracy compared to SC with OR-based accumulation and an up to $3.6\times $ improvement in energy compared to SC with binary addition. With proper modeling, our approach improves training performance for SC-based neural networks and makes training SC models practical for large datasets like ImageNet.

I. Introduction

Machine learning has become ubiquitous in fields, such as image recognition, voice recognition, and machine translation [1], [2], [3]. Deep learning using neural networks has been dominant in machine learning because of its ease of training and generalization capabilities [4]. Neural networks have grown in complexity in the past few years [1], [2], [4], with storage and compute requirements of deep learning outpacing hardware improvement [5], [6]. This increased cost makes it hard to run deep neural networks on resource-constrained devices. Stochastic computing (SC) aims to improve deep learning efficiency [7], [8]. The compact computation unit of SC promises reduced computation area, improved data reuse, and reduced memory access costs [3]. Since SC requires only single gates for multiplication and addition, it enables a large number of multiply–accumulate units on the same hardware, which greatly improves operator reuse and alleviates memory bottlenecks of deep learning accelerators [3].