Accurate Whole-Brain Segmentation for Bimodal PET/MR Images Via a Cross-Attention Mechanism

The PET/MRI system plays a significant role in the functional and anatomical quantification of the brain, providing accurate diagnostic data for a variety of brain disorders. However, most of the current methods for segmenting the brain are based on unimodal MRI and rarely combine structural and functional dual-modality information. Therefore, we aimed to employ deep learning techniques to achieve automatic and accurate segmentation of the whole brain while incorporating functional and anatomical information. To leverage dual-modality information, a novel 3D network with a cross-attention module was proposed to capture the correlation between dual-modality features and improve segmentation accuracy. Moreover, several deep learning methods were employed as comparison measures to evaluate the model performance, with the Dice similarity coefficient (DSC), Jaccard index (JAC), recall, and precision serving as quantitative metrics. Experimental results demonstrated our advantages in whole-brain segmentation, achieving an 85.35% DSC, 77.22% JAC, 88.86% recall, and 84.81% precision, which were better than those comparative methods. In addition, consistent and correlated analyses based on segmentation results also demonstrated that our approach achieved superior performance. In future work, we will try to apply our method to other multimodal tasks, such as PET/CT data analysis.