A CNN Bi-LSTM based Multimodal Continuous Hand Gesture Recognition

Multimodal gesture recognition is one of the important tasks in the field of Human-Computer Interaction. This paper proposes a Convolutional Neural Network-Bidirectional Long Short-Term Memory network based multimodal gesture recognition technique for continuous hand gestures. The modalities like RGB modality, optical flow modality and the segmented hand data are used as input to the feature extractor network. Apart from these modalities, this paper proposes the feature extraction from temporal differences of RGB modality. The pretrained Googlenet Caffe model is used to learn the features from these modalities. These features are concatenated for performing the fusion process. For the classification task, the Bidirectional Long Short-Term Memory network is used. The proposed method is tested on the IPN Hand dataset. Initially each modality is used individually. Thereafter their combinations are used for further improvement in classification accuracy. The combination of segmented hand modality, optical flow modality and temporal difference of RGB modality provided the best result with a Levenshtein accuracy of 62.22%. The simulation results show that the proposed approach outperforms the benchmark techniques for multimodal gesture recognition.