ListenBot: Augmented Reality Based Speech To Sign Language Conversion

Hearing impairment poses substantial challenges for individuals, hindering their effective communication and interaction within society. This paper addresses these challenges through the development of an application tailored for people with hearing disabilities. The application serves as a pivotal tool by retrieving audio input and transforming it into dynamic sign language animations, thereby facilitating seamless communication. The system captures spoken language and translates it into expressive sign language animations. This transformative approach not only enhances the communication abilities of individuals with hearing impairments but also fosters a more inclusive environment for them. The underlying technology incorporates Convolutional Neural Networks (CNN) for precise speech classification, ensuring accurate recognition of spoken words. In addition, the model employs Recurrent Neural Networks (RNN) to capture contextual speech, enhancing the authenticity and clarity of the sign language representation. The paper compares the results of different models generated using LSTM, GRU and machine learning methods consisting HMM and GMM. The final application is crafted using Unity using AR applications, guaranteeing a user-friendly interface that promotes accessibility. By seamlessly translating spoken language into visually comprehensible sign language, this application empowers individuals with hearing impairments, fostering a more inclusive and connected society.