Intelligent Framework for Auditory and Speech Impaired Individuals by Accelerating Sign Image Analysis using Graph Data Science

The prevalence of auditory and speech impairment in India is a significant public health concern, affecting a huge number of individuals across the country. The existing body of research believes that sign language is directly equivalent to spoken language, ignoring the linguistic aspects of sign language. Therefore, an inventive end-to-end system for translating and recognising sign language is required in order to facilitate real-time communication between users of sign language and those who do not. By integrating Sign Language Recognition (SLR) and Sign Language Translation (SLT) techniques, the system incorporates a deep learning network that analyses images for detection after being trained on a sizable dataset of hand gestures, and landmark estimation using the MediaPipe library, thus ensuring accurate real-time recognition of sign gestures. The SLT module utilizes graph data science and NetworkX to construct a directed weighted graph of Indian Sign Language (ISL) gestures and their corresponding English translations, facilitating the translation of gesture sequences into coherent English sentences. The study highlights the effectiveness and potential of combining SLR and SLT approaches using CNN is a dependable communication aid that enables people with speech and hearing problems to converse fluently with non-sign language users. Also, an extensive evaluation of a comprehensive dataset consisting of 56,200 images demonstrates outstanding performance and an impressive accuracy of 98.64% in recognizing and translating gestures.