Phrase Level Bangla Sign Language Recognition using Keypoints from Hand Gesture Video

Many people around the world are incapable of speaking. Sign language is the only means they use for conducting conversations. Phrase level sign language is the efficient way for hand gesture-based sign language communication. Hand detection and tracking is the crucial part for recognition of hand gesture-based sign. In research works conducted for hand gesture-based sign recognition, skin color segmentation, Kinect sensor-based data, specialized colored gloves for each hand palms etc. have been used. These research works had limitations. In this work, a dataset for recognition of 27 Bangla sign phrases was created. MediaPipe [1] library of Google was used to detect and track hands and extract key points from palm portion of each hand. Angle features between each key points of the hand palm were calculated. Long Short-Term Memory (LSTM) model was used for classification. Overall accuracy for Bangla sign language recognition by the proposed system was 92.07%. The system was also experimented on PkSLMNM [2] dataset, which is a Pakistan phrase level sign language dataset having 7 different sign phrases. Overall accuracy for recognition of phrases from PkSLMNM dataset by the proposed system was 90.98%, which was higher than the overall accuracy score of 82.66% [3] proposed by the authors of the PkSLMNM. Though the proposed system was tested on phrase level sign language recognition, it can be used in recognition of any hand gesture based sign and Human Computer Interaction (HCI).