Context Free Grammar(CFG)-based English/Urdu Translation System from Video & Audio to Pakistan Sign Language (PSL) Gestures

The communication barrier between hearing impaired and unimpaired populations presents a profound challenge, impacting millions in the deaf community. Despite numerous advancements in assistive technologies, translating natural languages to Pakistan Sign Language(PSL) remains significantly underdeveloped, exacerbating communication difficulties for those with hearing impairments. This research introduces an innovative multi-stage methodology designed to bridge this gap by translating English and Urdu media content into PSL gestures using a proposed Context-free-grammar (CFG) for high grammatical precision. Our approach begins with preprocessing audio streams to ensure clarity and compatibility with subsequent processing stages. We employ the Google Speech and NLP APIs to transcribe and process media content. The identified transcription is then processed using advanced machine learning algorithms to enhance its coherence and readability, punctuating and structuring it appropriately. We then tokenize the sentences and pass it to our English PSL module which employs tools to parse and manipulate input string using PSL grammar rules. Each word of the translated sentences is matched with the HamNoSys database and the SiGML tags are generated for each symbol and then exported to the SiGML player to animate the gesture on the user interface. The system’s performance was rigorously evaluated, demonstrating good reliability with an accuracy of 95.58% for sentence structures and 93.98% for tenses. The transformative potential of this work lies in enhancing accessibility and fostering inclusivity for the deaf community in Pakistan. By providing a robust real-time translation framework, our system addresses critical needs in education, healthcare, and employment.