A Virtual Sign Language Translator on Smartphones

Sign languages are natural languages used primarily by the deaf communities. Unfortunately, it is difficult for people outside of the communities to understand the languages. As smartphones are now ubiquitous, it will be very convenient for people from different communities to communicate if smartphones can help translate the languages. In this paper, a smartphone application called Sign Language Translator (SLT) is introduced, which records the person signing and translates the hand gestures into texts. SLT integrates two pre-trained models proposed by previous works and a classification model developed in this study. Given a frame showing the person making the sign, SLT first gets the body skeleton of the person, from which the positions of the hands are identified and the fingers' skeleton is then detected. The classification model is next used to extract features from the skeleton to map to the specific words that are expressed. In the experiments, we chose three specific words in American Sign languages for evaluation. Both training and testing datasets are the videos with the corresponding signs. The accuracy of translation into right words in the testing dataset is more than 80 percent. It shows that SLT can effectively help people to communicate with deaf people.