Real-Time Two Way Communication System for Speech and Hearing Impaired Using Computer Vision and Deep Learning

Sign Language is the most expressive form of communication for speech and hearing impaired people to communicate with normal person but a normal person cannot understand sign language. So in order to break this barrier of communication there needs to be a system that can enable conversion of sign language to voice or text and voice or text to sign language and do it in real time. The systems that currently exist are not real time, do not facilitate two-way communication, require static surrounding conditions or have low recognition accuracy. There exist systems that have good accuracy but require external hardware like gloves [3] which increases the cost. Our contribution to solving this problem consists of a Sign Language Communication System. It is a real-time communications system built using the advancements in Image Processing, Deep Learning and Computer Vision that provides real-time sign language to text and text to sign language conversion. The project is software-based which can be installed on any computer with good specifications. It is also a two-way communication system allowing not just speech and hearing impaired to communicate with normal people but also other way around. The primary goal of our system is to enable hearing and speech impaired people to communicate with people that are not disabled in real time by interpreting alphabets, numbers and words in the Indian sign language. The system is able to predict 17600 test images in 14 seconds with an average prediction time of 0.000805 seconds with an accuracy of 99%.