Online Action Detection by Long Short-term Transformer with Query Exemplars—transformer

The online action detection task is to recognize actions that occur in real-time in streaming video. Current works model historical dependencies and predict the future to sense action changes in video streams and improve recognition accuracy. Existing works have largely ignored category-level modeling. In this paper, we present a method, named Long Short-Term Transformer approach with query exemplars, which enables temporal dependency modeling of long and short-term historical memories in a unified framework and provides more information to guide temporal dependency modeling through category-level modeling. In addition, we emphasize the importance of short-term memory for current action recognition. Background suppression loss is also incorporated in our approach to further reduce false recognition of background frames that are very similar to action frames. We extensively validate our method on three benchmark datasets, achieving better results in recognition accuracy on THUMOS, TVseries, and HACS Segment. The experiments demonstrate that the introduction of category-level modeling and background suppression loss has a significant improvement effect on the accuracy of online action recognition.