Few-Shot Semantic Segmentation Based on Dual-Branch Feature Extraction

Few-shot semantic segmentation (FSS) requires only few labeled samples to achieve good segmentation performance and thus has received extensive attention. However, existing FFS methods usually adopt a simple convolutional structure as the backbone, which suffers from poor feature extraction ability. In order to address this issue, a novel few-shot segmentation network based on dual-branch feature extraction (DFESN) is proposed. First, an attention-enhanced ResNet is used as the local feature extraction branch. Specifically, we in-corporate channel attention operations into each building block of ResNet to model the importance among channels, which enables DFESN to learn important class information for the segmentation task. Besides, we introduce a Vision Transformer as the global feature extraction branch. This branch leverages the multi-head self-attention mechanism in Vision Transformer to model the global dependencies of support and query image features, further enhancing the feature extraction capabilities of DFESN. We conduct experiments on the PASCAL-5ⁱ dataset and demonstrate the superiority of our DFESN.

I. Introduction

As a basic problem in the field of computer vision, the purpose of semantic segmentation is to predict the pixel-wise class of images. FCN replaces the convolutional layers in CNNs with fully connected layers, and utilizes upsampling operations to restore image resolution. Later, U-Net adopts an encoder-decoder structure to connect multi-level features, and enriches the input of convolutional layers by skip connections. Recently, DeepLab V3+ explores a common feature fusion strategy in object detection, which improves segmentation speed while improving segmentation performance. Although these methods have made some progress in semantic segmentation, they need to manually label each training sample and cannot segment unseen categories. Therefore, some researchers try to investigate few-shot semantic segmentation (FSS) methods.