Sign Language Detection Using Faster RCNN Resnet

Sign language recognition unites society with people who cannot hear. Sign Language involves a series of hand gestures each with their own meaning and can vary with region and time. A system is required to overcome this barrier between the disabled people and others. Sign Language Recognition is classified in two different ways, they are image and sensor based. Image-based approaches for sign language recognition involve capturing multiple images of the signer performing a sign using one or more cameras, and then analyzing and processing these images to identify and recognize the sign being performed but are not perfectly accurate. In the sensor-based method, instrument gloves assembled with sensors are used to monitor the joints of the hands. These are very accurate but very costly and not easily accessible. In this project an image-based model is used to detect the signs made by a person. Steps involved are: Capturing an Image, preprocessing the image which includes resizing it to 640X640 size, feature extraction, sign classification. The model being used is Faster-RCNN- Resnet, a pretrained model from TensorFlow model zoo.

I. Introduction

Sign language expression is a way through which the disabled communicate, it gives voice to the voiceless. A study shows that there are around 466 million deaf people, out of which 34 million of them are adults, but 90% of them learn sign language from outside the family. It is hard to learn any language, sign language is no exception, and it is the only way to communicate with deaf people. Therefore, here comes the need for a model that can understand the disabled people and to become their voice. The formatter must produce these elements while taking into account the relevant requirements listed below. Deaf and mute persons find it incredibly challenging to communicate, thus we need a linguistic framework to decipher what they are attempting to say. Between silent persons and regular people, language serves as a barrier. Our goal is to develop an mediating surface that translates sign language into text so that everyone can comprehend the sign language of non-verbal persons. Sign language detection is the recognizing and interpreting gestures of hand and movements used in sign language. People who are disabled with deafness or hard of hearing use sign language, a visual language, to communicate with one another, as well as with people who can hear but also know sign language or expression. The detection involves use of computer vision and machine learning techniques to analyze video footage of sign language users, with the goal of translating the gestures and movements into text or spoken language. Sign language detection has a vast range of applications, including in education, communication, and accessibility. It can be used to make it efficient for hearing people to converse with people who are deaf and people, as well as to enable sign language interpretation in real-time situations such as meetings and lectures. Through this project the barrier between the deaf and others can be broken. We will be focused to train a model to identify hand motions for this reason. KNN, SVM, logistic regression, and CNN may all be used to train the model. Yet, CNN is more reliable and efficient when it comes to picture identification, which is why we used CNN for the training of our model. We can analyze with reasonable efficiency after the systematic model is trained using the ASL images, however the model does not give its full functionality using certain libraries like OpenCV to identify finger movements. In order to enhance the trained model, we built our user-specified dataset for training using the data containing photographs that were taken, and then predicted the text of signs using webcam input. After backdrop removal, combine the obtained images to black and white as well. This will enable more accurate hand gesture recognition. In this project a faster RCNN resnet, a pretrained model modified to accommodate the projects needs is used, where it is trained on 10 classes each of which consists of words like (Hello, Help, Hurt, Love You, No, One, Phone, Please, Thank You, Yes). In this project the objectives met were: To create a system that is effective at translating Sign language conversion to text using (DNN) and other libraries. A system which is efficient, accurate and cheap and easily accessible for the better communication between the disabled and society.