Cloud-Assisted Multiview Video Summarization Using CNN and Bidirectional LSTM

The massive amount of video data produced by surveillance networks in industries instigate various challenges in exploring these videos for many applications, such as video summarization (VS), analysis, indexing, and retrieval. The task of multiview video summarization (MVS) is very challenging due to the gigantic size of data, redundancy, overlapping in views, light variations, and interview correlations. To address these challenges, various low-level features and clustering-based soft computing techniques are proposed that cannot fully exploit MVS. In this article, we achieve MVS by integrating deep neural network based soft computing techniques in a two-tier framework. The first online tier performs target-appearance-based shots segmentation and stores them in a lookup table that is transmitted to cloud for further processing. The second tier extracts deep features from each frame of a sequence in the lookup table and pass them to deep bidirectional long short-term memory (DB-LSTM) to acquire probabilities of informativeness and generates a summary. Experimental evaluation on benchmark dataset and industrial surveillance data from YouTube confirms the better performance of our system compared to the state-of-the-art MVS methods.

I. Introduction

The tremendous amount of video data generated by camera networks installed at industries, offices, and public places meet the requirements of Big Data. For instance, a simple multiview network with two cameras, acquiring video from two different views with 25 frames per second (fps), generates 180 000 frames (90 000 for each camera) for an hour. The surveillance networks acquire video data for 24 hours from multiview cameras, thereby making it challenging to extract useful information from this Big Data. It requires significant effort when searching for salient information in such huge-sized 60 × 60 video data. Thus, automatic techniques are required to extract the prominent information present in videos without involving any human efforts. In the video analytics literature, there exist several key information extraction techniques such as video abstraction [1], video skimming [2], and video summarization (VS) [3]. VS techniques investigate input video for salient information and create a summary in the form of keyframes or short video clips that represent lengthy videos. The extracted keyframes assist in many applications such as action and activity recognition [4], [5], anomaly detection [6], and video retrieval [7].