Robotic Telepresence Based on Augmented Reality and Human Motion Mapping for Interventional Medicine

Robotic telemedicine has a high potential to promote the development of future medicine. It enables the treatment of critical patients in geographically remote locations in time, and also particularly useful for avoiding infection risks during pandemics of infectious diseases. However, visual feedback from the patient’s side to the clinician’s side is often unintuitive due to a lack of depth information, occlusion, and viewing angles. This paper develops a robotic telepresence method based on augmented reality and human motion mapping for interventional medicine. An optical see-through head-mounted display (OST-HMD) is mounted at the distal end of a redundant manipulator, which can provide views from different directions, and the clinician can see through key anatomies and instruments located inside the patient body. The control of the manipulator is accomplished via human motion mapping, with both the end-effector and manifold of the human arm being mapped to the manipulator. A simple evaluation method is developed to assess the accuracy of head localization of the OST-HMD. Telepresence experiments are carried out to verify the effectiveness of the proposed method, with which the operator can see through a human airway phantom.