Finger-writing-in-the-air system using Kinect sensor

A novel Kinect based finger-writing character recognition system called finger-writing-in-the-air is proposed in this paper. This system allows people to input characters by writing in the air virtually. We propose a mixture model for hand segmentation, which takes advantage of depth, color, and motion information. This model helps overcome problems arisen with traditional vision-based methods like illumination variation or hand-face overlapping, also solves the fatal color-depth mismatch problem accompanied by Kinect. A dual-mode switching algorithm is presented for accurate fingertip detection. It is less sensitive to noise along the segmented hand contour and can handle various hand poses. The positions of fingertip are linked and reconstructed as inkless character strokes. The recovered trajectory is recognized by the state-of-the-art handwriting character recognition method. Experiments show that user can write freely in the air inputting Chinese characters, English letters (upper and lower case) and digits in real time with over 90% accuracy rate for the first five candidates.