Revisiting Deep Learning Parallelism: Fine-Grained Inference Engine Utilizing Online Arithmetic

In this paper, we revisit the parallelism of neural inference engines. In a departure from the conventional coarse-grained neuron-level parallelism, we propose a synapse-level parallelism by performing highly parallel fine-grained neural computations. Our method employs online Most Significant Digit First (MSDF) digit-serial arithmetic to enable early termination of the computation. Using online MSDF bit-serial arithmetic for DNN inference (1) enables early termination of ineffectual computations, (2) enables mixed-precision operations (3) allows higher frequencies without compromising latency, and (4) alleviates the infamous weights memory bottleneck. The proposed technique is efficiently implemented on FPGAs due to their concurrent fine-grained nature, and the availability of on-chip distributed SRAM blocks. Compared to other bit-serial methods, our Fine-Grained Inference Engine (FGIE) improves energy efficiency by ×1.8 while having similar performance gains.