Intuitive Robot Path Planning through Augmented Reality

In this paper, an approach for intuitive path planning of a collaborative robot (cobot) based on augmented reality (AR) is presented. For this purpose, a user interface is developed that offers practical functions to the operator for full motion control over a robot manipulator. A mixed reality head-mounted display (HMD) acts as an interface for the operator by displaying virtual content that allows interaction with the robotic system. Besides providing information about the state of the robot, including joint position, velocity, and the force exerted on the robot flange, the gripping force can be adjusted during gripping operations, and motion commands can also be sent to the robot controller. It includes motion commands in joint and Cartesian space, as well as intuitive path planning of the robot based on waypoints created by the operator. A virtual cursor is incorporated into the virtual robot model, with its position and orientation aligned to that of the tool center point (TCP) of the real robot. The operator can execute linear point-to-point as well as non-linear motion. Regarding the safety aspects of the robot motion, a virtual model of the robot is also overlaid on the working environment, allowing the operator to carry out a preview motion to the given trajectory, showing the current and final position of each joint, before executing the motion on the real system. While the human-robot interface and virtual content are created on top of the Unity3D game engine, the Robot Operating System (ROS) provides reliable data transmission and processing between the HMD and the robot controller. The novelty of this work lies on the one hand, in an extensive interface and, on the other hand, in the robot path planning that takes into account the orientation provided by a virtual pointer.