Multimodal Deep Learning Approaches to Breast Tumor Characterization using Ultrasound B-Mode and Nakagami Parametric Images

This study proposes a multimodal deep learning architecture for breast tumor characterization that takes as input both ultrasound B-Mode and ultrasound Nakagami parametric images in the form of stacked 2D data-maps. Ultrasound data from 831 patients, among which 582 had benign and 249 had malignant lesions, was used in the study to train, validate and test the networks. The multimodal architecture obtained statistically significantly higher AUC, sensitivity and accuracy compared to networks taking an input of B-Mode or Nakagami image data by themselves. The obtained results suggest that B-Mode and Nakagami images contain complimentary information that is valuable for breast tumor characterization if utilized together in a deep learning framework and demonstrates the potential of this approach in improving breast cancer diagnosis.