Applying mmWave Radar Sensors to Vocabulary-Level Dynamic Chinese Sign Language Recognition for the Community With Deafness and Hearing Loss

To facilitate human–computer interaction (HCL) for the community with deafness and hearing loss (D&HL), this article explored the feasibility of recognizing a vocabulary of dynamic Chinese sign language (CSL) based on millimeter-wave (mmWave) radar sensors within the scope of data science. Fundamental problems that challenge its applications on computers and other electronic devices were addressed from multidisciplinary aspects including how to capture signs with an mmWave radar sensor, how to characterize signs using mmWave, and how to recognize signs based on the extracted features. Accordingly, this article proposed tentative solutions for the key concerns on this topic. A case study was later given by constructing a lightweight attentive augmented convolutional neural network (CNN) to classify 15 Chinese sign words based on radar spectrograms. The network achieved a 98.82% weight reduction and a 3.7% accuracy improvement over the original ResNet-18. Furthermore, a deep convolutional generated adversarial network for data augmentation was used to alleviate the conflicts between large numbers of parameters and small data sample sizes. This article shows the keenness of the authors to witness more practical developments in this new area with interdisciplinary collaboration and evolutionary process.

I. Introduction

World Health Organization announced in March 2020 that more than 5% of the global population uses sign languages [1]. As one of the most intrinsic ways of conveying semantic information out of the brain, sign language can further improve fluent communication for the community with deafness and hearing loss (D&HL) and help them be highly integrated into the global world. Many approaches have been proposed and studied by capturing signs with different commercial on-the-shelf sensors. Meanwhile, rapid developments in visual and wearable sensors facilitate the translation of sign language into the textual comprehension of computers and other electronic devices in the Internet-of-Things (IoT) era. In 2015, Tubaiz et al. [2] used data gloves to recognize 40 short sentences containing 80 words in Arabic sign language and combined with an optical camera to segment and align hand movements with their corresponding sign words with a sentence-level recognition rate higher than 98%. Wu et al. [3] proposed a wearable real-time American sign language recognition (SLR) system in 2016, which recognized 80 isolated American sign language (ASL) words by capturing hand and arm movements with an inertial measurement unit and surface emulsion signals. The recognition results of the same and different volunteers reached 96.16% and 85.24%, respectively. Koller et al. [4] proposed a new approach to large vocabulary continuous SLR across multiple signers in 2015 and formed a publicly available large vocabulary database that contributed to the development of SLR. Later, Ibrahim et al. [5] proposed a computer vision-based SLR system in 2017 and tested it on isolated Arabic words with a recognition accuracy of 97%. Chong and Lee [6] used Leap Motion controller in 2018 to classify 10 digits, 26 letters, and 36 sign words in American sign language (ASL) based on finger and hand movements with a recognition accuracy of up to 72.79%. Despite years of development, there are still some problems in the research aforementioned: contact wearable inertial sensors result in unnatural hand movement due to the wire and circuitry worn on hands. Computer vision-based SLRs depend on optical cameras, but their applications are subject to environmental darkness and potential privacy breaches. Therefore, finding new alternatives to achieve intelligent, efficient, and accurate human–computer interaction (HCI) for the D&HL community becomes significant.