A Bifocal Classification and Fusion Network for Multimodal Image Analysis in Histopathology

Recognition of key morphological features in histological slides is crucial for pathological diagnosis and monitoring therapeutic progress. However, the typical routine microscopic workflow is conducted by hand which is time-consuming and has unavoidable intra- and inter-observer variability like all human work. Therefore, we propose a bifocal classification and fusion network for the automated recognition and cross-modality analysis of diagnostic features in whole-slide multimodal images (WSIs). In brief, paired image tiles cropped from digitized tissue sections were fed into a modified dual-path CNN which accepts asymmetric inputs for classification, and then the inference results were converted to feature distribution heatmaps, which permit qualitative as well as quantitative morphological analyses of entire histological sections, even in combination with adjacent sections that have been stained differently. The multimodal heatmaps were aligned using image registration and fused for cross-modality analysis. Our experiments showed that the network achieved high recognition performance (AUCs of 0.985 and 0.988, and accuracies of 94.7% and 96.1% on two WSI modalities, respectively, against expert markings) and outperformed state-of-the-art methods without training on a large cohort or utilizing domain transfer. In addition, the new method involves a self-contained inference and fusion process and thus harbors significant potential for speeding up microscopic analysis workflows.