Efficient Video Object Co-Localization With Co-Saliency Activated Tracklets

Video object co-localization is the task of jointly localizing common visual objects across videos. Due to the large variations both across the videos and within each video, it is quite challenging to identify and track the common objects jointly. Unlike the previous joint frameworks that use a large number of bounding box proposals to attack the problem, we propose to leverage co-saliency activated tracklets to efficiently address the problem. To highlight the common object regions, we first explore inter-video commonness, intra-video commonness, and motion saliency to generate the co-saliency maps for a small number of selected key frames at regular intervals. Object proposals of high objectness and co-saliency scores in those frames are tracked across each interval to build tracklets. Finally, the best tube for a video is obtained through selecting the optimal tracklet from each interval with the help of confidence and smoothness constraints. Experimental results on the benchmark YouTube-objects dataset show that the proposed method outperforms the state-of-the-art methods in terms of accuracy and speed under both weakly supervised and unsupervised settings. Moreover, by noticing the existing benchmark dataset lacks of sufficient annotations for object localization (only one annotated frame per video), we further annotate more than 15k frames of the YouTube videos and develop a new benchmark dataset for video co-localization.

I. Introduction

Localizing primary objects [1], [2] in videos is an important task in computer vision, since it facilitates many other vision tasks such as object recognition, retrieval and action recognition. Following the success of joint processing research in images [3]–[5], recent research interests have been shifted from single-video object localization to video object co-localization [6], [7], which aims at jointly localizing common objects across videos by exploiting shared attributes among videos as a type of weak supervision.