Autonomous Recognition of Multiple Surgical Instruments Tips Based on Arrow OBB-YOLO Network

In surgery, the robot can automatically adjust the endoscope to ensure the stability of the surgical image and reduce the burden on the doctor. It is becoming a research hotspot of current surgical assistance. The identification and localization of endoscopic instruments tips is the key to achieve efficient “doctor–robot” collaboration. However, traditional methods have the problems of poor real-time performance and insufficient information expression. This article proposes an autonomous recognition of multiple surgical instrument tips based on arrow object bounding box (OBB)-YOLO network prediction, the proposed method not only realizes the recognition and localization of key parts of the instrument, but also uses arrows to enhance the expression of critical information such as the tip angle under the premise of meeting the real-time requirements for medical scenes. First, based on the modified YOLOv3 model, the Arrow OBB-YOLO network used to predict Arrow OBB is built, and the arrows are constrained inside the bounding box (BB) through coordinate transformation. Second, the optimization expression of arrow errors can enhance the prediction effect of the model, and the designed step-by-step training optimization method significantly accelerates the convergence speed of the model. Then, based on the Arrow OBB model, a method for generating the tracking areas is proposed. Finally, the experimental results show that the proposed method can accurately identify and locate the tips of the instrument with 36.5 fps and can improve the recognition effect of the BB when adding new information expression, which lays the foundation for “doctor–robot” collaborative surgery.