A real time system for dynamic hand gesture recognition with a depth sensor

Recent advances in depth sensing provide exciting opportunities for the development of new methods for human activity understanding. Yet, little work has been done in the area of hand gesture recognition which has many practical applications. In this paper we propose a real-time system for dynamic hand gesture recognition. It is fully automatic and robust to variations in speed and style as well as in hand orientations. Our approach is based on action graph, which shares similar robust properties with standard HMM but requires less training data by allowing states shared among different gestures. To deal with hand orientations, we have developed a new technique for hand segmentation and orientation normalization. The proposed system is evaluated on a challenging dataset of twelve dynamic American Sign Language (ASL) gestures.

1. INTRODUCTION

Hand gestures are commonly used for human-human communications. We do not even think about it when we use hand gestures because they feel so natural to us. It is a natural question whether the hand gesture could become one of the main modalities for human-computer interactions. Driven by this elusive goal, tremendous amount of research activities have been devoted to hand gesture recognition. So far, most of the research work has been focused on recognition from images and videos due to the popularity of digital cameras. The recent progress in depth sensors such as Microsoft's Kinect device has generated a new level of excitement in gesture recognition. With the depth information, a skeleton tracking system has been developed by Microsoft. Many game developers have leveraged the skeleton tracking system to design games with body gestures as the main interaction mode.