A Data Driven Approach for Computer Vision based Sign Language Model using Convolutional Neural Network

Human beings communicate with each other with the help of speech. However, not everyone on our planet is endowed with the ability to talk and hear. People with such difficulties in speech and hearing, cannot communicate with each other by normal means. So, they use set of signs and hand gestures to communicate with each other. The set of such hand signs and gestures is known as sign language. Yet, they cannot communicate with normal people as most people out there do not know how to use sign language. The available methods are applied only on the level of alphabets, making them useless for live time usage. The primary goal of this research work is to convert sign language into speech on world level with the help of computer vision, tensorflow and natural language processing by extracting features from hand signs along the features from facial emotions as facial emotions also play vital role in the world level sign language. The hand signs and gestures along with the facial emotion features are converted into text with the help of an object detection model supported by tensorflow and computer vision. The artificial intelligence techniques - convolution neural network has been used for the training and predicting the speech input feed. The text output from the object detection model is then converted into speech with the help of pyttsx package, which is a renowned python package in the field of natural language processing.

I. Introduction

Human beings, the most socio-intelligent creatures, communicate with each other with the help of speech. But nearly one of fourteen people in this world were not blessed with the ability to speak and hear. They have difficulties with speech or hearing. By normal means the normal people will find it difficult to communicate with People having such difficulties in speech. So, to communicate with each other they use signs and hand gestures called as sign language. After detailed browsing about sign language, it is found that there are about 240 sign languages exist for spoken languages in the world. As most of the normal people out there do not know how to use sign language they cannot communicate with people having such disability. The already available methods are applied only on the level of alphabets which made them useless for real time usage. The goal of this initiative is to increase communication with people who have hearing and speech impairment by using sign language to express themselves [3] [4].