An End-to-end Anchorless Approach to Recognize Hand Gestures using CenterNet

Hand gesture recognition is one of the interesting problems of Computer Vision. It has a wide range of applications in the fields of Human-Computer Interaction, Robotics, Sign language interpretation, Augmented Reality, etc. Most of the existing deep learning methods detect hand gestures in two stages. The hand is located in the first stage, and classification is performed on the hand portion in the second stage to estimate the hand pose. Although these methods are accurate, they are slow and cant be used for real-time applications. Few existing literature even explored one-stage approaches, like YOLO, SSD, etc., for hand gesture recognition as they have less inference time. But they place many anchor boxes over an image of which only a small percentage are positive. This leads to a huge imbalance between positive and negative anchor boxes and slows the training process. In this paper, we have used an end-to-end, one-stage hand detection-based approach, namely, CenterNet, for hand gesture recognition. It detects the object as a point, i.e., the center point of the bounding box encompassing the object, and regresses to the object size. This eliminates the need for anchor boxes in CenterNet. We have added Dual Attention Network to the CenterNet architecture to improve the performance. Our model achieves a mean F1-score of 84.40% and 98.83% on Ouhands and NUS hand pose datasets, respectively. Results show that our model can perform well even under complex backgrounds and varying illumination conditions, and the F1-scores obtained are close to benchmark values.