Discovering Primary Objects in Videos by Saliency Fusion and Iterative Appearance Estimation

In this paper, we propose a new method for detecting primary objects in unconstrained videos in a completely automatic setting. Here, we define the primary object in a video as the object that presents saliently in most of the frames. Unlike previous works considering only local saliency detection or common pattern discovery, the proposed method integrates the local visual/motion saliency extracted from each frame, global appearance consistency throughout the video, and spatiotemporal smoothness constraint on object trajectories. We first identify a temporal coherent salient region throughout the whole video, and then explicitly learn a global appearance model to distinguish the primary object against the background. In order to obtain high-quality saliency estimations from both appearance and motion cues, we propose a novel self-adaptive saliency map fusion method by learning the reliability of saliency maps from labeled data. As a whole, our method can robustly localize and track primary objects in diverse video content, and handle the challenges such as fast object and camera motion, large scale and appearance variation, background clutter, and pose deformation. Moreover, compared with some existing approaches that assume the object is present in all the frames, our approach can naturally handle the case where the object is present only in part of the frames, e.g., the object enters the scene in the middle of the video or leaves the scene before the video ends. We also propose a new video data set containing 51 videos for primary object detection with per-frame ground-truth labeling. Quantitative experiments on several challenging video data sets demonstrate the superiority of our method compared with the recent state of the arts.

I. Introduction

With the prevalence of online social video sharing, considerable amounts of videos are being created and processed every day. In many of those videos, there exists a primary object that we want to focus our attention on, e.g., a child or a pet in a homemade personal video. We define the primary object in a video sequence as the object that presents saliently in most of the frames, and some examples are shown in Fig. 1. In this paper, we address the problem of automatically discovering the primary objects in videos, which is an essential step for many applications such as advertisement design [36] and video summarization [20], [28], [44]. Traditional video object detection and localization methods, however, are either too category specific (e.g., face [47] and pedestrian detection [13]) or heavily rely on manual initialization (e.g., object tracking [19] and interactive object segmentation [18]). They are suitable for targeted object detection that is tailored to users’ interests, but are too limited for many multimedia applications that require automatically processing large volumes of video data with diverse content. Throughout this paper, we will also use the term foreground object or simply foreground interchangeably with the term primary object. Fig. 1.

Examples of primary object discovery. Each row corresponds to one video, and the red rectangle highlights the primary object.