Adapting Segment Anything Model for Change Detection in VHR Remote Sensing Images

Vision foundation models (VFMs), such as the segment anything model (SAM), allow zero-shot or interactive segmentation of visual contents; thus, they are quickly applied in a variety of visual scenes. However, their direct use in many remote sensing (RS) applications is often unsatisfactory due to the special imaging properties of RS images (RSIs). In this work, we aim to utilize the strong visual recognition capabilities of VFMs to improve change detection (CD) in very high-resolution (VHR) RSIs. We employ the visual encoder of FastSAM, a variant of the SAM, to extract visual representations in RS scenes. To adapt FastSAM to focus on some specific ground objects in RS scenes, we propose a convolutional adaptor to aggregate the task-oriented change information. Moreover, to utilize the semantic representations that are inherent to SAM features, we introduce a task-agnostic semantic learning branch to model the semantic latent in bitemporal RSIs. The resulting method, SAM-based CD (SAM-CD), obtains superior accuracy compared with the state-of-the-art (SOTA) fully supervised CD methods and exhibits a sample-efficient learning ability that is comparable to semisupervised CD methods. To the best of our knowledge, this is the first work that adapts VFMs to CD in VHR RSIs.

I. Introduction

Recently, vision foundation models (VFMs) have emerged and gained great research focus in the field of computer vision. Utilizing the knowledge gained in meta-scale datasets, foundation models, such as segment anything model (SAM) [1] and its variants (e.g., FastSAM [2] and Mobile SAM [3]) are able to recognize visual contents in a training-free manner and obtain fine-grained semantic masks. Their strong generalization across different imaging conditions and visual objects promotes greatly the applications in real-world scenarios. However, due to the inductive bias learned in natural images, foundational models exhibit limitations when applied to images in some specific domains, such as medical images and remote sensing images (RSIs). According to the literature survey [4], SAM pays more attention to the foreground objects and often fails to segment small and irregular objects. In this article, we focus on adapting SAM to improve one of the fundamental tasks in RS, i.e., change detection (CD) of very high-resolution (VHR) RSIs.