A Real-Time Gesture Recognition System with FPGA Accelerated ZynqNet Classification

This paper presents a real-time hand gesture recognition system by accelerating a convolutional neural network (CNN) using FPGA platform. More specifically, ZynqNet is adopted and modified to fulfill the classification task of recognizing the Swedish manual alphabet, which is used by sign language users for spelling purposes, also known as fingerspelling. Data augmentation and transfer learning techniques have been used during the training phase to improve the classification accuracy up to 80.1%, even with an 8-bit ZynqNet model. Extensive analysis of memory requirements and data processing patterns has been performed to enable optimization techniques, including memory partitioning and register arrays. The resulting FPGA implementation on a Xilinx UltraScale device avoids the use of off-chip memories, which together with block-wise processing scheduling, achieves an image rate of 23.5 frames per second (FPS) at 200 MHz clock frequency.