A Bi-directional Interactive System of Sign Language and Visual Speech Based on Portable Devices

At present, the natural communication between deaf and normal people is a major challenge both in theoretical research and application. In addition, in the field of Human-Machine Interaction, there is little work on bi-directional communication based on sign language and visual speech. In this paper, a portable bi-directional interactive system of sign language and visual speech was proposed to help deaf people communicate naturally under certain circumstances. In the section of modeling, Network in Network model was employed to classify sign language words, while the method using a network combined DenseNet and LSTM together was used for lip reading. By wearing the portable device, people can achieve bi-directional translation and communication of visual speech and sign language. We built our own Chinese sign language database containing more than 100 categories of words and conducted experiments in the context of the airport for experimental verification. Eventually, the average time of each round of conversation is within 15 seconds, faster than normal communication without system assistance, which verified the effectiveness and advantage of the purposed system.