Control with Gestures: A Hand Gesture Recognition System Using Off-the-Shelf Smartwatch

Smart wearable devices are becoming smaller, cheaper and popular. Smartwatch is one of the most popular wearable devices. The smartwatch has rich applications such as messages, email and voice by connecting to smartphone via Bluetooth. It is hard to interact with the smartwatch due to the small screen and the way it is worn. Since the smartwatch is usually equipped with sensors like accelerometer and gyroscope and is worn on the wist, which makes it possible to identify user's gestures by tracking the movement of user's finger, hand and arm. Furthermore, the user can control other nearby smart devices with gestures if they can be identified by smartwatch via bluetooth. In this paper, we propose a system that collect, process and store and classify smartwatch built-in sensors data. We propose an effective action detection and segment algorithm, and show that the start of actions can be detected and well segmented from continuous action data. We further propose a bidirectional LSTM based deep neural network model which can recognize the length-variant action data in the accuracy as high as 96%. We implement a smartwatch app together with an android smartphone app that can inference gestures in real time. Our system shows the potential of interacting with smart devices like smartwatch and wrist-band and use them as a remote controller.