A Real-Time and Hardware-Efficient Processor for Skeleton-Based Action Recognition With Lightweight Convolutional Neural Network

Skeleton-based human action recognition (HAR) has been extensively studied these years because body skeleton has the simple but informative representation of human action, which greatly reduces the computation complexity compared with the image-based HAR. As a result, it is suitable for low power implementation in embedded platforms. In this brief, we present a systematic approach to developing a hardware-efficient and low-power processor for real-time skeleton-based HAR. First, a lightweight HAR algorithm only using the one-dimensional convolutional neural network (1D-CNN) is proposed. Second, the singular value decomposition is employed to compress the weights in the fully connected (FC) layers of the proposed convolutional neural network. Third, a hardware processor implementing the proposed algorithm is presented. Aimed at optimizing area and energy, this processor utilizes a flexible structure supporting different kernel sizes of the 1D-CNN and reuses hardware in both convolution layers and FC layers. The proposed processor is implemented under SMIC 65-nm CMOS technology and consumes a total area of 1.016 mm². Experimental results show that the proposed processor can achieve state-of-the-art classification accuracy in NTU RGB+D dataset and SBU dataset while outperforming previous solutions in area and energy efficiency.

I. Introduction

Body skeleton gains great attention [1]–[4] recently due to its simple but informative representation of body movement. Action recognition based on skeleton does not suffer from the context noise [5] compared with image-based methods [6], [7] so that it is tempting for scientific research.