Circulating Genetically Abnormal Cells Identification Based on Improved Mask R-CNN and Signal Detection Model

Among the current methods for early diagnosis of lung cancer, circulating genetically abnormal cells (CAC) testing is a non-invasive or minimally invasive, convenient and costeffective method for early tumor diagnosis. Fluorescence in situ hybridization (FISH) is a method that helps to detect abnormal copies of specific genes in CAC. Although fluorescence in situ hybridization has high accuracy in diagnosing lung cancer, it is difficult to accurately identify CACs due to nuclear overlap. In this study, we proposed an end-to-end FISH-based method for CAC identification. The method achieves nuclear segmentation and computes four-color staining signals by adding a mask header to the cascade structure of the cascade R-CNN, and then classifies the cell categories (normal cells, missing cells, gain cells, or CAC) according to pathological criteria. In addition, the segmentation accuracy of overlapping cell nuclei is further improved by adding an edge constraint algorithm to the training process. We trained and tested the proposed model on a dataset containing 700 frames and 58,083 cell nuclei, and the Intersection over Union (IoU), Dice coefficient, and Hausdorff distance for cell nucleus segmentation were $93.8 \%, 95.6 \%$, and $97.84 \%$, respectively. The accuracy, precision and AP of signal point detection are better than other models. The results indicate that the proposed method is superior to existing CAC recognition methods and is a promising method for early screening of lung cancer.