DC²T: Disentanglement-Guided Consolidation and Consistency Training for Semi-Supervised Cross-Site Continual Segmentation

Continual Learning (CL) is recognized to be a storage-efficient and privacy-protecting approach for learning from sequentially-arriving medical sites. However, most existing CL methods assume that each site is fully labeled, which is impractical due to budget and expertise constraint. This paper studies the Semi-Supervised Continual Learning (SSCL) that adopts partially-labeled sites arriving over time, with each site delivering only limited labeled data while the majority remains unlabeled. In this regard, it is challenging to effectively utilize unlabeled data under dynamic cross-site domain gaps, leading to intractable model forgetting on such unlabeled data. To address this problem, we introduce a novel Disentanglement-guided Consolidation and Consistency Training (DC2T) framework, which roots in an Online Semi-Supervised representation Disentanglement (OSSD) perspective to excavate content representations of partially labeled data from sites arriving over time. Moreover, these content representations are required to be consolidated for site-invariance and calibrated for style-robustness, in order to alleviate forgetting even in the absence of ground truth. Specifically, for the invariance on previous sites, we retain historical content representations when learning on a new site, via a Content-inspired Parameter Consolidation (CPC) method that prevents altering the model parameters crucial for content preservation. For the robustness against style variation, we develop a Style-induced Consistency Training (SCT) scheme that enforces segmentation consistency over style-related perturbations to recalibrate content encoding. We extensively evaluate our method on fundus and cardiac image segmentation, indicating the advantage over existing SSCL methods for alleviating forgetting on unlabeled data.

I. Introduction

Recently, Convolutional Neural Network (CNN)-based methods have achieved promising performance in medical image segmentation, however, they heavily rely on extensive supervised data collected from multiple medical sites (i.e., hospitals) for model training [1]. Nevertheless, acquiring these multi-site datasets simultaneously and aggregating them into a unified set is impractical due to high storage costs and privacy concerns [2]. It is desirable to use an incremental data stream for model training, i.e., Continual Learning (CL), where the model is encouraged to learn consecutively from sequentially-arriving sites without catastrophic forgetting on the previously-learned yet currently-inaccessible sites [3]. Despite its success, existing CL typically performs in the fully-supervised training setting, which is extremely expensive to manually annotate all data in each arriving site. Yet, in clinical practice, most medical sites often only afford partial data labeling, as it saves financial budgets and circumvents the high demand for annotators with comprehensive expertise [4]. How to leverage these partially-labeled medical sites in CL is a realistic question to be solved.