Integral Object Mining via Online Attention Accumulation

Object attention maps generated by image classifiers are usually used as priors for weakly-supervised segmentation approaches. However, normal image classifiers produce attention only at the most discriminative object parts, which limits the performance of weakly-supervised segmentation task. Therefore, how to effectively identify entire object regions in a weakly-supervised manner has always been a challenging and meaningful problem. We observe that the attention maps produced by a classification network continuously focus on different object parts during training. In order to accumulate the discovered different object parts, we propose an online attention accumulation (OAA) strategy which maintains a cumulative attention map for each target category in each training image so that the integral object regions can be gradually promoted as the training goes. These cumulative attention maps, in turn, serve as the pixel-level supervision, which can further assist the network in discovering more integral object regions. Our method (OAA) can be plugged into any classification network and progressively accumulate the discriminative regions into integral objects as the training process goes. Despite its simplicity, when applying the resulting attention maps to the weakly-supervised semantic segmentation task, our approach improves the existing state-of-the-art methods on the PASCAL VOC 2012 segmentation benchmark, achieving a mIoU score of 66.4% on the test set. Code is available at https://mmcheng.net/oaa/.

1. Introduction

Benefiting from the large-scale pixel-level training data and advanced convolutional neural network (CNN) architectures, fully-supervised semantic segmentation approaches, such as [4 , 20 , 22 , 42 , 38] , have made great progress recently. However, constructing a large-scale pixel-accurate dataset is fairly expensive and requires considerable human efforts and time cost. In order to economize human labors, researchers propose to learn semantic segmentation using weak supervision, such as bounding boxes [27] , points [2] , and even image-level annotations [26] . Among these weak supervisions, image-level annotations can be more easily obtained than other annotations. Thus, in this paper, we focus on semantic segmentation under image-level supervision. Figure 1.

Observation of our proposed approach. (a) Source images; (b-d) Intermediate attention maps produced by a classification network at different training stages; (e) Cumulative attention maps produced by combining attention maps in (b), (c), and (d) through a simple element-wise maximum operation. It can be easily observed that the discriminative regions continuously shift over different parts of the semantic objects. The fused attention maps in (e) can record most of semantic regions compared to (b), (c), and (d). Best viewed in color.