FASN: Feature Aggregate Side-Network for Open-Vocabulary Semantic Segmentation

In this paper, we introduce an Feature Aggregate Side Network (FASN), a simple, efficient, and easy-to-train method for open-vocabulary semantic segmentation. Building upon existing models based on the CLIP-Side Network framework, we address the issue of CLIP-generated features lacking pixel-level recognition capability by layering a novel fast graph representation learning layer between the CLIP and side networks. This integration introduces an inductive bias for the aggregation of local features, thereby better addressing the challenges in semantic segmentation. Through validation on five distinct datasets and extensive ablation studies, we have demonstrated the effectiveness of our modifications. Our findings indicate that with a slight increase in the number of parameters, there is a significant enhancement in performance.