Exploring the Use of Deep Reinforcement Learning Algorithms for Wound-Approaching Trajectories in Robot-Assisted Minimally Invasive Surgery

Robot-assisted minimally invasive surgery (RAMIS) has improved the quality of operations, execution time, and patient recovery time. However, suturing remains a tedious, complex, and time-consuming task. To automate this procedure, we have investigated the use of Deep Deterministic Policy Gradient (DDPG) to generate a trajectory for Wound-Approaching (WA). The trajectory must satisfy surgical constraints, be neither too long nor too wide, and minimize recovery time and side effects. We compare standard DDPG, DDPG from Demonstration (DDPGfD), and we have developed a new algorithm that learns from success epochs, DDPG from Success (DDPGfS). An ad hoc reward function was developed to minimize the number of actions and avoid undesirable behaviour in the neighbourhood of the wound. Among the algorithms, DDPGfS proves to be the best choice in terms of reliability and robustness with a percentage of success equal to 74% respect to 64% of DDPGfD. The new dense reward function enhanche the performance, either in low support situation, preventing the agent to oscillate around the entry point.