GlandSAM: Injecting Morphology Knowledge Into Segment Anything Model for Label-Free Gland Segmentation

This paper presents a label-free gland segmentation, GlandSAM, which achieves comparable performance with supervised methods while no label is required during its training or inference phase. We observe that the Segment Anything model produces sub-optimal results on gland dataset: It either over-segments a gland into many fractions or under-segments the gland regions by confusing many of them with the background, due to the complex morphology of glands and lack of sufficient labels. To address this challenge, our GlandSAM innovatively injects two clues about gland morphology into SAM to guide the segmentation process: (1) Heterogeneity within glands and (2) Similarity with the background. Initially, we leverage the clues to decompose the intricate glands by selectively extracting a proposal for each gland sub-region of heterogeneous appearances. Then, we inject the morphology clues into SAM in a fine-tuning manner with a novel morphology-aware semantic grouping module that explicitly groups the high-level semantics of gland sub-regions. In this way, our GlandSAM could capture comprehensive knowledge about gland morphology, and produce well-delineated and complete segmentation results. Extensive experiments conducted on the GlaS dataset and the CRAG dataset reveal that GlandSAM outperforms state-of-the-art label-free methods by a significant margin. Notably, our GlandSAM even surpasses several fully-supervised methods that require pixel-wise labels for training, which highlights the remarkable performance and potential of GlandSAM in the realm of gland segmentation.

I. Introduction

Label-free gland segmentation refers to the process of segmenting a whole slide image (WSI) into a glandular region and a non-glandular region without the use of any kind of labels during the training and inference phases. Despite the success of the existing studies on fully supervised gland segmentation [1], [2], [3], [4], [5], [6] and weakly supervised gland segmentation [7], [8], these approaches necessitate pixel-level or weaker labels, e.g., bound-box and patch tag. However, the manual labeling of WSIs remains a considerable challenge due to their extensive scale [9]. Specifically, it usually takes months for a pathology expert to draw pixel-level labels for one WSI at the resolution of [9], while weaker forms of annotations, e.g., bound box, still cost more than three weeks [8]. To tackle this challenge, in this paper, we propose the first work of label-free gland segmentation, which enables training and inference without relying on any explicit labels.