Selecting Suitable Data Input for Deep-Learning Sign-Language Recognition with a Small Dataset

Deep learning-based sign-language recognition usually requires abundant training videos. This research considers generating valid sign-language data for training the recognition models. MediaPipe is used to acquire the hand skeleton from the sign-language video. Then we analyze several hand skeleton adjustment policies with color-weighting strategies and generate hand masks to simulate different hands. Since miss detections of hands may happen due to motion blurring caused by rapid hand movements, we incorporate optical flows to ensure that the hand movement information is retained in each frame. We employ different spatial and temporal data augmentation strategies to simulate varying hand sizes, filming angles, and hand speeds. The experimental results show that the proposed method improves the accuracy of sign-language recognition in the American Sign Language dataset.