Unified Multi-Modal Image Synthesis for Missing Modality Imputation

Multi-modal medical images provide complementary soft-tissue characteristics that aid in the screening and diagnosis of diseases. However, limited scanning time, image corruption and various imaging protocols often result in incomplete multi-modal images, thus limiting the usage of multi-modal data for clinical purposes. To address this issue, in this paper, we propose a novel unified multi-modal image synthesis method for missing modality imputation. Our method overall takes a generative adversarial architecture, which aims to synthesize missing modalities from any combination of available ones with a single model. To this end, we specifically design a Commonality- and Discrepancy-Sensitive Encoder for the generator to exploit both modality-invariant and specific information contained in input modalities. The incorporation of both types of information facilitates the generation of images with consistent anatomy and realistic details of the desired distribution. Besides, we propose a Dynamic Feature Unification Module to integrate information from a varying number of available modalities, which enables the network to be robust to random missing modalities. The module performs both hard integration and soft integration, ensuring the effectiveness of feature combination while avoiding information loss. Verified on two public multi-modal magnetic resonance datasets, the proposed method is effective in handling various synthesis tasks and shows superior performance compared to previous methods.

I. Introduction

Multi-modal medical images are widely adopted in disease screening and diagnosis due to their ability to provide complementary soft-tissue characteristics and diagnostic information. For instance, commonly acquired magnetic resonance (MR) sequences include T1-weighted, T2-weighted, post-contrast T1-weighted (T1Gd), and fluid-attenuated inversion recovery (FLAIR) images, each of which is considered as a distinct modality that highlights specific anatomy and pathology. Clinically, a combination of multiple modalities is often used to present pathological changes and assist clinicians in making accurate diagnoses. However, obtaining complete multi-modal images for each patient can be challenging due to factors such as limited scanning time, motion or artifact-induced image corruption, and the use of different imaging protocols [1]. When dealing with incomplete data, it is undesirable to simply discard it as it often contains valuable information, and also infeasible to re-scan missing sequences for data completion due to the high cost of data acquisition. Therefore, multi-modal image synthesis (also known as data imputation) has been explored to generate missing modalities from limited available data, which has the potential to benefit downstream data analysis (i.e., segmentation [2], registration [3]), enhance the diagnostic accuracy of diseases (i.e., Alzheimer’s disease [4]), and assist surgery planning [5].