A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction

In this paper, we propose HF²-VAD, a Hybrid framework that integrates Flow reconstruction and Frame prediction seamlessly to handle Video Anomaly Detection. Firstly, we design the network of ML-MemAE-SC (Multi-Level Memory modules in an Autoencoder with Skip Connections) to memorize normal patterns for optical flow reconstruction so that abnormal events can be sensitively identified with larger flow reconstruction errors. More importantly, conditioned on the reconstructed flows, we then employ a Conditional Variational Autoencoder (CVAE), which captures the high correlation between video frame and optical flow, to predict the next frame given several previous frames. By CVAE, the quality of flow reconstruction essentially influences that of frame prediction. Therefore, poorly reconstructed optical flows of abnormal events further deteriorate the quality of the final predicted future frame, making the anomalies more detectable. Experimental results demonstrate the effectiveness of the proposed method. Code is available at https://github.com/LiUzHiAn/hf2vad.

1. Introduction

Video Anomaly Detection (VAD) refers to the identification of events that do not conform to expected behaviors [3] in a video, with one example shown in Figure 1. This is an open and very challenging task as abnormal events usually much less happen than normal ones and the forms of abnormal events are unbounded in practical applications [25]. Obviously, it is impossible to collect all kinds of abnormal data in advance. Therefore, a typical solution to video anomaly detection is to train an unsupervised learning model on normal data, and those events or activities that are recognized by the trained model as outliers are then deemed as anomalies.