VLUT: Design and Evaluation of Variable band LUT to realize Activation Functions

In this work two novel approaches to design hardware accelerated activation functions are proposed and evaluated against few pre-existing methods mentioned in the literature. These approaches along with new methods were compared based on the accuracy, resources utilized, critical path delay, and the power consumed. The different methods that are discussed and evaluated to approximate the activation functions are: (i) a classical approach with single-fold lookup tables (LUTs) to approximate the activation functions, (ii) a Two-fold LUT consisting of data and error compensation separately with equal band sizes for the overall input range, (iii) a proposed two-fold LUT with variable band size, also referred to as variable two-fold LUT (v-LUT). (iv) a piece-wise-linear (PWL) approximation with fixed point representation, and (v) a new method for implementing the activation functions using floating-point IP block with single precision 32-bit representation. The Two-fold LUT with variable band size also referred to as v-LUT offered compact spacing and minimal usage of memory resources among all the designs, whereas 32-bit single precision floating-point IP method prowided the most accurate results. The proposed Two-fold with v-LUT outperforms the existing literature of Two-fold LUT approach in terms of hardware metrics for activation functions such as Sigmoid, tanh, Softsign, and $\frac{x}{\sqrt{\left(1+x^{2}\right)}}$ without compromising on the accuracy.