Hand Gesture Recognition Using MediaPipe and CNN for Indian Sign Language and Conversion to Speech Format for Indian Regional Languages

In recent times, deep learning techniques have made remarkable progress across various domains and applications. However, Indian Sign Language (ISL) recognition, audio generation translation, still present significant challenges from a developmental perspective. In this paper, introduction of a novel method to create a broad framework for real-time ISL recognition, translation tasks. To enhance recognition accuracy, leveraged the power of the MediaPipe library and employ a hybrid Convolutional Neural Network model for extracting pose details and generating text. On the other hand, for producing sign gesture audio corresponding to spoken sentences, used GTTs, supporting the conversion of the audio into various Indian Regional Languages like Hindi, Kannada, Telugu, Marathi, etc. The proposed mechanism tackles the complexities present in earlier approaches and achieves an impressive accuracy of about 94%. Extensively tested the mechanism during its development phases, and evaluation metrics demonstrate convincing improvements over previous methods. In conclusion, this novel approach, combining MediaPipe, CNN, has significantly improved the recognition and production of sign language gestures. The model's performance on evaluation metrics showcases its potential to enhance communication and accessibility for individuals with hearing and speech impairments.