TSO-DETR: A Network for Small Object Detection of Cervical Cells in TCT Smear

Cervical cell detection, a crucial component in Thin-prep Cytologic Test (TCT) screening for early cervical cancer diagnosis, faces challenges due to the small size of abnormal cells. To address this issue, we propose a novel model, the TCT Small Object DEtection TRansformer (TSO-DETR). TSO-DETR incorporates two key techniques to improve detection performance. Firstly, a multi-scale approach is employed by integrating PVTv2 and Deformable DETR. This approach enhances the feature representation of small cervical cells by combining shallow and deep features from different scales. Secondly, a novel approach is utilized by combining Spatial Reconstruction Unit (SRU) with our proposed Mixture Channel Attention (MixCA). This combination enhances the feature extraction ability of the model from both spatial and channel dimensions. Notably, the introduction of MixCA enables the model to prioritize important image features, thereby reducing interference from background noise. Comparative experimental results demonstrate TSO-DETR’s superiority over other state-of-the-art methods in our TCT dataset, achieving 24.8% in mean Average Precision (mAP) and 50.5% in mean Average Recall (mAR). These results reflect improvements of 8.0 percentage points in mAP and 10.6 percentage points in mAR, respectively, over the second-best model. Consequently, TSO-DETR can serve as an additional valuable method for cervical cell detection. The source code is available at https://github.com/threedteam/TSO-DETR/.