Scale-Aware Graph Neural Network for Few-Shot Semantic Segmentation

Few-shot semantic segmentation (FSS) aims to segment unseen class objects given very few densely-annotated support images from the same class. Existing FSS methods find the query object by using support prototypes or by directly relying on heuristic multi-scale feature fusion. However, they fail to fully leverage the high-order appearance relationships between multi-scale features among the support-query image pairs, thus leading to an inaccurate localization of the query objects. To tackle the above challenge, we propose an end-to-end scale-aware graph neural network (SAGNN) by reasoning the cross-scale relations among the support-query images for FSS. Specifically, a scale-aware graph is first built by taking support-induced multi-scale query features as nodes and, meanwhile, each edge is modeled as the pairwise interaction of its connected nodes. By progressive message passing over this graph, SAGNN is capable of capturing cross-scale relations and overcoming object variations (e.g., appearance, scale and location), and can thus learn more precise node embeddings. This in turn enables it to predict more accurate foreground objects. Moreover, to make full use of the location relations across scales for the query image, a novel self-node collaboration mechanism is proposed to enrich the current node, which endows SAGNN the ability of perceiving different resolutions of the same objects. Extensive experiments on PASCAL-5ⁱ and COCO-20ⁱ show that SAGNN achieves state-of-the-art results.

1. Introduction

Deep convolutional neural networks [14],[29],[40] have advanced the development of many downstream vision tasks, such as semantic segmentation [1], [3], [21], [25], [49], which is a dense prediction task. To achieve an effective segmentation model, e.g. Deeplab [3], large amounts of pixel-wise image annotations are required for model training, which are costly and time consuming to acquire. Weakly supervised learning [23], [43] can alleviate the annotation costs to some extent; however, the model performance drops significantly under this scenario. Moreover, both fully and weakly supervised models suffer poor generalization to unseen domains with only a few densely-annotated training images available. As such, few-shot semantic segmentation (FSS) [27] is proposed for dealing with the unseen object segmentation and data annotation issue.