Real Time Detection and Conversion of Gestures to Text and Speech to Sign System

In their daily lives, Deaf-Mute people face numerous challenges when performing simple tasks. One of the many challenges is the ability to communicate with others through sign language. Unfortunately, only 0.2 percent of the population uses and practices ASL (American Sign Language). This divides Deaf-Mute people from those who want to communicate with them. The proposed "Sign Language Translator" system aims to bridge this gap, and it is hoped that it will help Deaf-Mute people converse with people who are unfamiliar with ASL. The system can be used to help non-native speakers learn the language. The goal of the system is to convert speech to American Sign Language by using an artificial neural network known as RNN (Recurrent Neural Network) with LSTM (Recurrent Neural Network - Long Short-Term Memory) trained by a Connectionist Temporal Classification (CTC) neural network. This paper additionally presents gesture-based communication to text/speech model that enables a method for laying out a two-way correspondence without the need of an interpreter. For gesture recognition this paper uses the SSD Mobilenet V2 model.