Remembering history with convolutional LSTM for anomaly detection

This paper tackles anomaly detection in videos, which is an extremely challenging task because anomaly is unbounded. We approach this task by leveraging a Convolutional Neural Network (CNN or ConvNet) for appearance encoding for each frame, and leveraging a Convolutional Long Short Term Memory (ConvLSTM) for memorizing all past frames which corresponds to the motion information. Then we integrate ConvNet and ConvLSTM with Auto-Encoder, which is referred to as ConvLSTM-AE, to learn the regularity of appearance and motion for the ordinary moments. Compared with 3D Convolutional Auto-Encoder based anomaly detection, our main contribution lies in that we propose a ConvLSTM-AE framework which better encodes the change of appearance and motion for normal events, respectively. To evaluate our method, we first conduct experiments on a synthesized Moving-MNIST dataset under controlled settings, and results show that our method can easily identify the change of appearance and motion. Extensive experiments on real anomaly datasets further validate the effectiveness of our method for anomaly detection.

1. Introduction

Anomaly detection is an important task in computer vision on account of its prevalent applications in video surveillance, video summarization, and scene understanding, etc. However, this task remains extremely challenging because it is an ill-posed problem, i.e., the scenarios of abnormal event are unbounded because it is extremely difficult or infeasible to collect data corresponding to all abnormal events. In contrast, the acquisition of ordinary moments in videos is much easier. Thus, a common setting for anomaly detection is that there are only ordinary moments available in the training sets.¹

†: Equal Contribution

Anomaly detection can be casted as the following two subproblems: i)how to characterize the appearance and motion; ii)how to model the change in appearance or motion. For quite a long time, hand-crafted features [1] [2] are utilized to characterize the appearance and motion in videos, then sparse representation based approaches [3]–[5] can be used to measure the change of appearance or motion. However, such sparse representation strategy is very time-consuming for both training and testing. Recently, deep neural networks have shown their advantages over hand-crafted features for visual data representation in image classification [6] and activity recognition [7] [8]. Recently, Hasan et al. [9] propose to use a 3D Convolutional Neural Network (ConvNet or CNN) based Auto-Encoder framework to simultaneously learn the regularity among the appearance and motion for anomaly detection. However, many existing work for activity recognition have shown that 3D convolution is not good enough for motion characterization [10] [11].