Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition

Conventional 3D convolutional neural networks (CNNs) are computationally expensive, memory intensive, prone to overfitting, and most importantly, there is a need to improve their feature learning capabilities. To address these issues, we propose spatio-temporal short-term Fourier transform (STFT) blocks, a new class of convolutional blocks that can serve as an alternative to the 3D convolutional layer and its variants in 3D CNNs. An STFT block consists of non-trainable convolution layers that capture spatially and/or temporally local Fourier information using an STFT kernel at multiple low frequency points, followed by a set of trainable linear weights for learning channel correlations. The STFT blocks significantly reduce the space-time complexity in 3D CNNs. In general, they use 3.5 to 4.5 times less parameters and 1.5 to 1.8 times less computational costs when compared to the state-of-the-art methods. Furthermore, their feature learning capabilities are significantly better than the conventional 3D convolutional layer and its variants. Our extensive evaluation on seven action recognition datasets, including Something$^2$ v1 and v2, Jester, Diving-48, Kinetics-400, UCF 101, and HMDB 51, demonstrate that STFT blocks based 3D CNNs achieve on par or even better performance compared to the state-of-the-art methods.

1 Introduction

In recent years, with the availability of large-scale datasets and computational power, deep neural networks (DNNs) have led to unprecedented advancements in the field of artificial intelligence. In particular, in computer vision, research in the area of convolutional neural networks (CNNs) has achieved impressive results on a wide range of applications such as robotics [1], autonomous driving [2], medical imaging [3], face recognition [4], and many more. This is especially true for the case of 2D CNNs where they have achieved unparalleled performance boosts on various computer vision tasks such as image classification [5], [6], semantic segmentation [7], and object detection [8].