Attention-Augmented Electromagnetic Representation of Sign Language for Human-Computer Interaction in Deaf-and-Mute Community

In order to provide a new interface between computers and deaf-and-dumb users, this paper proposed a method of translating sign language into a sequence of time-frequency spectrograms based on a 24 GHz 1T-2R Doppler radar sensor. By processing two pairs of the immediate frequency I/Q signals based on time-frequency analysis, a complete sign sentence can be captured and segmented according to the electromagnetic wave-based patterns. Rather than the traditional classifier, a convolutional neural network was utilized to classify the basic signs and make the complete sentence lucid to the computer. For greater accuracy, an attention module was augmented to the network. The proposed methods could reach the accuracy of 96% in translating short sentences such as “Yes”, “No”, “Thanks”, and “Hello”, which are with the highest usage rate in sign language. The work done by this paper can be considered as a supplement to current human-computer interactions, especially for the deaf-and-dumb community.

I. Introduction

Sign language is currently a primary communication manner in deaf-and-mute communities worldwide, more than 460 million. Translating varieties of signs into the patterns acknowledged by computers can be significant in improving access to the latest electronic products for the deaf-and-mute community. As a visual language, sign language contains hand gestures, body movements, and even signers’ facial expressions. The state-of-the-art work of sign language translation has been extensively investigated by using wearable devices like gloves and mechanomyogram signals [1] . These methods focus on capturing the features of the movement but are less user comfort for signers. Computer vision based [2] sign translation is touchless and could recognize both statistic signs and dynamic signs by image and video. However, the accuracy is heavily impacted in a low light environment. Therefore, our motivation in this paper is to allow deaf-and-mute community to have a new touchless way of translating their sign language into the compatible text form of the current electronic devices. To our knowledge, electromagnetic (EM) wave extensively investigated in touchless hand gesture recognition can overcome the drawback of low light and reduce the risk of privacy leakage. A Doppler radar sensor can emit EM wave into space and receive echo signals reflected by the target. The representation of signs can be achieved by processing the micro-Doppler signatures from the radar echo signals to characterize different sign motions. Therefore, this paper discusses the feasibility of representing sign language based on a new pattern, EM wave.