Prototypical Metric Segment Anything Model for Data-Free Few-Shot Semantic Segmentation

Few-shot semantic segmentation (FSS) is crucial for image interpretation, yet it is constrained by requirements for extensive base data and a narrow focus on foreground-background differentiation. This work introduces Data-free Few-shot Semantic Segmentation (DFSS), a task that requires limited labeled images and forgoes the need for extensive base data, allowing for comprehensive image segmentation. The proposed method utilizes the Segment Anything Model (SAM) for its generalization capabilities. The Prototypical Metric Segment Anything Model is introduced, featuring an initial segmentation phase followed by prototype matching, effectively addressing the learning challenges posed by limited data. To enhance discrimination in multi-class segmentation, the Supervised Prototypical Contrastive Loss (SPCL) is designed to refine prototype features, ensuring intra-class cohesion and inter-class separation. To further accommodate intra-class variability, the Adaptive Prototype Update (APU) strategy dynamically refines prototypes, adapting the model to class heterogeneity. The method's effectiveness is demonstrated through superior performance over existing techniques on the DFSS task, marking a significant advancement in UAV image segmentation.

I. Introduction

Signal processing from Unmanned Aerial Vehicles' (UAV) perspective [1], [2], [3] is of significant importance in areas such as urban planning, agricultural management, and military security. Although existing generic CNN-based [4] and ViT-based [5] semantic segmentation methods can achieve remarkable results on UAV images, which are contingent upon extensive annotated datasets, the complexity of UAV imagery poses a challenge to the annotation process, making it both labor-intensive and costly. The few-shot semantic segmentation (FSS) is conceived to mitigate the limitations posed by a scarcity of samples. However, existing methods [6], [7], [8], [9] heavily rely on extensive base-training and focus mainly on distinguishing between the foreground and background, limiting their applicability to UAV images where a broader range of categories is needed. Consequently, this study introduces a segmentation approach for UAV imagery that operates with a limited dataset, leveraging the robust generalization capabilities of the Segment Anything Model (SAM) [10]. Specifically, an innovative few-shot semantic segmentation method is proposed, grounded in a foundation model and prototypical matching, tailored for a more demanding and pragmatic context. This method, termed Data-free Few-shot Semantic Segmentation (DFSS), transcends the necessity for extensive base datasets, enabling the comprehensive categorization of all discernible classes within UAV imagery.