A Configurable FPGA Implementation of the Tanh Function Using DCT Interpolation

Efficient implementation of non-linear activation functions is essential to the implementation of deep learning models on FPGAs. We introduce such an implementation based on the Discrete Cosine Transform Interpolation Filter (DCTIF). The proposed interpolation architecture combines simple arithmetic operations on the stored samples of the hyperbolic tangent function and on input data. It achieves almost 3× better precision than previous works while using a similar amount computational resources and a small amount of memory. Various combinations of DCTIF parameters can be chosen to trade off the accuracy and the overall circuit complexity of the tanh function. In one case, the proposed architecture approximates the hyperbolic tangent activation function with 0.004 maximum error while requiring only 1.45 kbits BRAM memory and 21 LUTs of a Virtex-7 FPGA.