SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation

One-shot image semantic segmentation poses a challenging task of recognizing the object regions from unseen categories with only one annotated example as supervision. In this article, we propose a simple yet effective similarity guidance network to tackle the one-shot (SG-One) segmentation problem. We aim at predicting the segmentation mask of a query image with the reference to one densely labeled support image of the same category. To obtain the robust representative feature of the support image, we first adopt a masked average pooling strategy for producing the guidance features by only taking the pixels belonging to the support image into account. We then leverage the cosine similarity to build the relationship between the guidance features and features of pixels from the query image. In this way, the possibilities embedded in the produced similarity maps can be adopted to guide the process of segmenting objects. Furthermore, our SG-One is a unified framework that can efficiently process both support and query images within one network and be learned in an end-to-end manner. We conduct extensive experiments on Pascal VOC 2012. In particular, our SG-One achieves the mIoU score of 46.3%, surpassing the baseline methods.

I. Introduction

Object semantic segmentation (OSS) aims at predicting the class label of each pixel. Deep neural networks have achieved tremendous success on the OSS tasks, such as U-net [1], FCN [2], and Mask R-CNN [3]. However, these algorithms trained with full annotations require many investments to expensive labeling tasks. To reduce the budget, a promising alternative approach is to apply weak annotations for learning a decent network of segmentation. For example, previous works have implemented image-level labels [4]–[6]; scribbles [7]–[9]; bounding boxes [10], [11]; and points [12]–[14] as cheaper supervision information whereas the main disadvantage of these weakly supervised methods is the lack of the ability for generalizing the learned models to unseen classes. For instance, if a network is trained to segment dogs using thousands of images containing various breeds of dogs, it will not be able to segment bikes without retraining the network using many images containing bikes.