MIFRL: A Model for Automatic Diagnosis of Laryngeal Diseases Based on Multi-Modal Information Fusion

Insufficient clinical experience and heavy workload may lead to misdiagnosis or omission when manually analyzing laryngoscope images and diagnosing diseases. Based on deep learning, automatic diagnosis and generation of endoscopic reports are of great importance. Therefore, based on both text and image modalities, we propose the multimodal internal-related feature representation learning (MIFRL) network model for multi-attribute recognition and endoscopic reports generation. Firstly, text and image encoding modules are used to extract feature, which simplifies the process of manually labeling images. Next, internal-related feature mining module is designed for multimodal information fusion and class imbalance problem. Finally, a multi-branch structure is designed to output the multi-attribute recognition of the lesions and generate endoscopic reports. In addition, asymmetric loss (ASL) is introduced to further mitigate the class imbalance problem. Based on our dataset, the average accuracy of endoscopic reports and attributes classification were 85.5% and 90.7%, respectively. In the recognition of the smallest number of leukoplakia images, mean average precision (mAP) was improved from 81.4% to 86.7%. Compared with other models, our model also achieved better results.