A Design of Inference Engine for Recurrent Neural Networks Using Block-Circulant Weight Matrices Based on a SoC-FPGA Approach

The inference engine design of recurrent neural networks (RNNs) is increasingly important for edge-AI applications. To improve the operation efficiency of an RNN inference engine with large weight matrix dimensions, previous works have presented the block-circulant weight matrix scheme that reduces the order of weight amounts for each
$n$
-dimensional matrix block from
$\mathrm{O}(n^{2})$
to
$\mathrm{O}(n)$
. This paper proposes an RNN inference engine design using a SoC-FPGA platform based on a SW-HW co-design scheme. In our design, the HW unit can perform individual gate operation involving the
$n$
-dimensional matrix-vector multiplication (
$n$
is the block size from 8 to 128) for
$m$
hidden neurons (
$m$
is a multiple of
$n$
), and the whole inference operation can be constructed through the SW. The proposed RNN inference engine was verified to demonstrate the phoneme classification using the TIMIT dataset. Compared to the previous FFT -based works, our approach lends support to more flexible (
$m,\ n$
) combinations for various RNN models.