Breast Imaging by Convolutional Neural Networks From Joint Microwave and Ultrasonic Data

Convolutional neural networks to achieve joint inversion of microwave and ultrasonic data for breast imaging are investigated. Source and field quantities, obtained via backpropagation, are used as inputs. A multistream structure is employed to benefit from data of different modalities. The network outputs the distribution maps of electric and acoustic parameters directly to achieve real-time imaging. Apart from the regression task, a multitask learning strategy is used with a classifier that associates each pixel to a tissue type to yield a segmentation image. Weighted loss is used to assign a higher penalty to pixels in tumors when wrongly classified. Comparisons are carried out between different network structures with the same datasets. The prediction results of the networks are evaluated by Intersection over Union for segmentation results and relative error of retrievals. The simulations on breast phantoms extracted from a dedicated repository show that, with both microwave and ultrasonic data, the network can provide a proper estimate of the breast structure and detection of small tumors. Meanwhile, multitask learning improves the regression results, and multistream input helps to exploit data from different modalities.