Semi-Supervised and Class-Imbalanced Open Set Medical Image Recognition

Open set recognition (OSR) for medical images is vital to ensure practical safeguards in clinical applications. It demands establishing model awareness of rare and unknown conditions, rejecting what the model does not know, and thus preventing these conditions from being dangerously misclassified into any false known classes. Meanwhile, it often further requires training the OSR model based on a mixture of severely class-imbalanced and partially labeled data due to the prevalence of long-tailed data distribution and the massive cost of medical image annotations. Unfortunately, most existing OSR methods perform poorly under such complex conditions, largely because they neglect to thoroughly exploit the knowledge from the imbalanced and unlabeled data to help better distinguish the known and unknown lesion patterns. This paper proposes a novel Semi-supervised and Class-Imbalanced Open Set medical image Recognition (SCI-OSR) algorithm as a joint solution to address these issues. The major contributions are as follows. Firstly, we propose a modified open mixup approach boasting selective fine-grained lesion foreground fusion based on the learned weighted attentive masks to augment the unlabeled, small-sized, and even pseudo-unknown class data, where the generated attentive masks serve as effective candidate lesion area estimators. Secondly, our approach can more effectively extract the disentangled and discriminative feature representations based on our tailored consistency and contrastive losses, which helps more properly exploit the augmented data and aids in pushing the learned features of known and unknown classes to be well separated by large margins. Finally, a prototype-based open set classifier is established based on the extracted discriminative features of the original and augmented data. It favors producing equally low confidence scores over all non-positive classes and thus more reliably rejects the unknown query data. Through comparative experiments and ablation studies conducted on several popular medical benchmarks, we demonstrate that our proposed SCI-OSR can successfully boost OSR performances, leading to improved known class classification and unknown class rejection over other state-of-the-art methods.