Real-Time Sign Language Recognition Based on Video Stream

There are millions of deaf-dumb people in the world communicating by sign language, thus designing a sign language recognition system is very meaningful and valuable for normal people to understand them. In this paper, we investigate a real-time Chinese sign language recognition system. A Chinese sign language dataset is firstly created. Considering practical applications, RGB camera is used to collect video stream, instead of RGB-D camera. In order to improve the accuracy of recognition, we propose a 3D-CNN method combined with optical flow processing. The collected RGB video stream is processed by optimized dense optical flow, and then put into 3D-CNN to extract feature vectors. For practical considerations, a real-time sign language recognition system is designed, composed of artificial interaction interface, motion detection module, hand and head detection module, etc. Experimental results show the superb performance and the applicability of the proposed systems.