Text Knowledge-Guided Segment Anything Model for Medical Image Segmentation

In this paper, we propose a method that utilizes clinical knowledge to bridge the domain gap when applying the Segment Anything Model (SAM) to medical images. Many recent methods employing SAM for medical image segmentation have focused on fine-tuning the model on medical images by integrating adaptation modules or modifying the framework. However, these approaches have limitations in understanding domain-specific knowledge. To overcome such limitations, we introduce a novel method that uses clinical knowledge of the target label. Specifically, our method includes a knowledge encoder that takes external knowledge on the anatomical structure to be segmented as input. The features extracted from the knowledge encoder enable the mask decoder to adapt to the target domain more effectively. Experimental results demonstrate that the performance of the SAM can be improved in the segmentation of medical images by the proposed method of injecting domain knowledge.