Sign Language Recognition using micro-Doppler and Explainable Deep Learning

In this paper, Sign Language Recognition and classification of the micro-Doppler signatures of different British Sign Language (BSL) gestures is studied. A database of four different BSL hand gesture motions is presented in the form of micro-Doppler signals, recorded with a continuous waveform radar. For detecting the presence of the micro-Doppler signatures, joint time-frequency is applied by calculating their spectrograms. Each individual gesture is expected to contain unique spectral characteristics that are exploited in order to classify the gestures. A deep learning approach with transfer learning is studied and discussed for carrying out the classification task. Following this, a novel explainable AI algorithm is implemented to give the user visual feedback, in the form of colour highlights, for the most relevant features used to classify each signal.

I. Introduction

Sign language is a way of communication using hand gestures and shapes, facial expressions and body movement that deaf people use. Although it follows a set of complex grammar rules and extensive vocabulary, it completely differs from the spoken an written language. In this work, BSL is examined.