Image-based Pose Estimation and Tracking of Surgical Instruments in Minimally Invasive Surgery

This study proposes a method for image-based surgical instrument pose estimation. In minimally invasive surgery (MIS), stereo vision can be used for surgical tool recognition and pose estimation. We use color markers to indicate the instrument's joints and detect them with a custom-made stereo camera to obtain their 3D coordinates. Kalman filter is adopted to attain the position of the given instrument's joints. We propose a kinematic algorithm to calculate the 6-Dof pose of the surgical instrument in the Cartesian coordinate system and the rotation angles of its joints. The proposed method is verified on a lab-built experimental platform, consisting of a 6-DoF HIWIN manipulator and a motorized surgical instrument. The experiment involves the instrument's tracking with a stereo camera in MIS pick-and-place routine. Experimental results show that the visual system has a maximum absolute error lower than 2.7 millimeters at the endpoint and the MAE of joint angles is less than 0.11 radians. The developed image-based pose estimation system can be used for the surgeon's training in MIS and may be applied to autonomous surgical operations.

I. Introduction

Minimally invasive surgery (MIS) is an advanced surgical treatment procedure, which has provided surgeons with a less invasive method of accessing the surgical site compared with open surgery. The advantages of MIS include smaller incisions, less bleeding and shorter recovery time, but it is more difficult to operate than open surgery. Laparoscopic instruments reduce the surgeon’s dexterity and the limited field of view of the surgical camera makes self-localization difficult. In addition to this, the learning curve for MIS is steep with surgeons taking significantly much time to master the technique. As a result, training of minimally invasive surgery is very important. Before conducting the real surgical operation, surgeons must go through a large number of related training [1] to ensure that they are proficient in MIS. Understanding every action in a surgical operation in real-time is with the potential to aid doctors in self-learning [2]. Actions that need improvement can be detected and corrected by the doctor himself.