Video Saliency Prediction Using Spatiotemporal Residual Attentive Networks

This paper proposes a novel residual attentive learning network architecture for predicting dynamic eye-fixation maps. The proposed model emphasizes two essential issues, i.e., effective spatiotemporal feature integration and multi-scale saliency learning. For the first problem, appearance and motion streams are tightly coupled via dense residual cross connections, which integrate appearance information with multi-layer, comprehensive motion features in a residual and dense way. Beyond traditional two-stream models learning appearance and motion features separately, such design allows early, multi-path information exchange between different domains, leading to a unified and powerful spatiotemporal learning architecture. For the second one, we propose a composite attention mechanism that learns multi-scale local attentions and global attention priors end-to-end. It is used for enhancing the fused spatiotemporal features via emphasizing important features in multi-scales. A lightweight convolutional Gated Recurrent Unit (convGRU), which is flexible for small training data situation, is used for long-term temporal characteristics modeling. Extensive experiments over four benchmark datasets clearly demonstrate the advantage of the proposed video saliency model over other competitors and the effectiveness of each component of our network. Our code and all the results will be available at https://github.com/ashleylqx/STRA-Net.

I. Introduction

Humans are able to rapidly orient attention to important areas in visual field and filter out irrelevant information. Such selective process, called as visual attention mechanism, helps humans operate huge amount of visual information in realtime. Visual attention has long been studied in computer vision community dated back to 1990s [1], and shown wide applications such as object segmentation [2], [3], [77], and video summarization [4], to name a few. Integrating attention mechanism into above tasks could allocate limited computation source into the most task-related targets, as well as obtain biologically inspired results.