Semantic Monocular Surgical SLAM: Intra-Operative 3D Reconstruction and Pre-Operative Registration in Dynamic Environments

In recent years, the feasibility of autonomy in the field of Robotic-Assisted Minimally Invasive Surgery (RAMIS) has been investigated. One of the most important requirements for such a system is the capability of reconstructing the patient's 3D anatomy in real-time and registering it with pre-operative data. This is a crucial step for surgical guidance and augmented reality applications. A common solution is to use Simultaneous Localization and Mapping (SLAM) which plays an important role in the field of computer vision and robotics. Conventional SLAM algorithms assume to operate in a static environment, which is clearly in contrast with the non-rigid and ever-changing nature of the surgical scene. In this paper, we propose a novel approach to monocular SLAM, combining geometric and semantic segmentation based on prior intervention-specific knowledge to automatically isolate dynamic objects and increase the accuracy of 3D reconstruction. Then, we register the sparse reconstructed point cloud with the preoperative anatomical model. The proposed approach is tested on an anatomically realistic phantom for partial nephrectomy with the da Vinci Research Kit (dVRK) endoscope. We compare our methodology with the state-of-the-art ORB-SLAM2 and perform an ablation study to assess the impact of semantic and geometric segmentation on the quality of reconstruction and registration.