Few-Shot Semantic Segmentation with Cyclic Memory Network

Few-shot semantic segmentation (FSS) is an important task for novel (unseen) object segmentation under the data-scarcity scenario. However, most FSS methods rely on unidirectional feature aggregation, e.g., from support prototypes to get the query prediction, and from high-resolution features to guide the low-resolution ones. This usually fails to fully capture the cross-resolution feature relationships and thus leads to inaccurate estimates of the query objects. To resolve the above dilemma, we propose a cyclic memory network (CMN) to directly learn to read abundant support information from all resolution features in a cyclic manner. Specifically, we first generate N pairs (key and value) of multi-resolution query features guided by the support feature and its mask. Next, we circularly take one pair of these features as the query to be segmented, and the rest N-1 pairs are written into an external memory accordingly, i.e., this leave-one-out process is conducted for N times. In each cycle, the query feature is updated by collaboratively matching its key and value with the memory, which can elegantly cover all the spatial locations from different resolutions. Furthermore, we incorporate the query feature re-adding and the query feature recursive updating mechanisms into the memory reading operation. CMN, equipped with these merits, can thus capture cross-resolution relationships and better handle the object appearance and scale variations in FSS. Experiments on PASCAL-5ⁱ and COCO-20ⁱ well validate the effectiveness of our model for FSS.

1. Introduction

Training high performance semantic segmentation models [1], [3], [18], [24], [42], based on convolutional neural networks [12], [27], [38], [47], typically requires large amounts of human-annotated training data, e.g., pixel-level annotations are essential for training a desirable segmentation model. However, data annotating by humans is usually costly and labor-intensive. Moreover, these models, almost always, fail to segment novel (unseen) objects, when given very few (one) training images (image) with annotations. To this end, as in conventional zero- and few-shot classification models [28],[36],[37] that aim to mitigate data annotation and novel object recognition issues in the high-level semantic category space, few-shot semantic segmentation (FSS) [25] has become an active research topic for alleviating these issues in the low-level image pixel space, under the object segmentation scenario.