Costa Rican Sign Language Recognition using MediaPipe

The launch of Google MediaPipe as a real-time platform for capturing and tracking the human body has allowed the removal of technological barriers in hand recognition for the construction of sign language translation software. It is of interest to understand how this platform can be used to encode Costa Rican Sign Language videos in low-dimensional structures to establish the basis for sign classification using machine learning. The experiment proposed in this paper targets the encoding of videos into one-dimensional vectors, the augmentation of video data to increase recognition capabilities, and the use of different machine learning and distance-based methods for sign recognition. The data for this experiment consists of 117 videos of one-second length (on average) from Costa Rican Sign Language (LESCO) emergency situations. This data format makes the encoding more challenging than performing finger spelling recognition, as each video is composed of many frames, and multiple hands can be outlined in each frame. The proposed approach allows the compression of all this information from a single video into a labeled vector added to a CSV file. The approach under evaluation recognized unary signs with an accuracy of 87.9% for the Extra Trees Classifier and 92.0% by using the Manhattan Distance.
¹