Design of CueAid: An Indian Sign Language Recognition Application for Differently Abled

The Indian Sign Language (ISL) is a widely used mode of communication among individuals with hearing and speech impairments in India. Unlike spoken language, sign language relies on visual cues such as hand gestures, arm posture, hand orientation, and facial expressions to convey a speaker's message. The majority of communication exchanges in today's society take place via video calls, which have been increasingly in demand. However, differently abled people find it challenging to communicate during video calls as it restricts their ability to express themselves. As a result, it is essential to convert sign language into a form that everyone can understand. The proposed application would help bridge the communication gap between the hearing and speech impaired community and the rest of society. In this study, we propose a system for recognizing action-based ISL. The designed application, CueAid, has two main functions: speech to gesture and gesture to text conversion. The system captures motion using OpenCV, identifies key points using MediaPipe, and employs a trained Long Short-Term Memory (LSTM) model for dynamic sign recognition. A custom dataset containing 15 everyday ISL words was created and used in the study, yielding an average accuracy of 93.334%. The created model has the potential to be integrated with the application for real-time recognition of ISL.