3D Knee Structure Reconstruction from 2D X-rays Based on Generative Deep Learning Models

In medical field, X-rays and Magnetic Resonance Imaging (MRI) are a valuable imaging modality. X-rays have the advantage of being more accessible than MRIs, but MRIs offer a much various information. In this study, we present a methodology for generating 3D MR images, with an em-phasis on capturing soft tissue features, alongside bones and discernible tissues, which are inadequately depicted in X-ray images. Therefore, we propose a deep learning model to generate 3D MR images from 2D X-rays by implementing generative deep learning models. The proposed model is divided into two stages: it performs image translation from (1) 2D X-rays to 2D MR images based on Generative Adversarial Networks (GAN) with cycle consistency in order to encompass features represented in both imaging modalities, and then generates (2) 3D MR images from the transformed 2D MR images in radiance field. As a result, our proposed model demonstrated superior performance compared to other models in both quantitative and qualitative evaluations. We confirm that our model successfully captures the characteristics of the knee depicted in X-rays and performs image transformation effectively, leading to outstanding performance in reconstructing them into 3D images. We expect that the proposed method will not only be utilized in real medical settings to achieve cost savings but also effectively assist healthcare specialists.