The remote operation and environment reconstruction of outdoor mobile robots using virtual reality

This paper presents a teleoperation system for controlling multiple semiautonomous mobile robots using the latest virtual reality (VR) technologies. Compared to direct traditional teleoperation of the robots (with a joystick and video display etc.), in this system the user operates the robots in a virtual world, where the real environment is reconstructed and streamed by the robots. The user can view and inspect the situation and collect data from a suitable perspective in VR as well as, for example, being able to enlarge or shrink himself for closer study. One of the biggest advantages of the developed system is that the user does not suffer from motion sickness generated due to latency in traditional teleoperations, where only a camera video stream is provided for the user directly. Also, with the implemented indirect control that uses the waypoints of each robot, one user can operate multiple robots at the same time as each robot is executing a defined path semiautonomous. The system described in this paper is a combination of the latest VR consumer devices, time-of-flight depth cameras, and adaptive network communication, used for scaling the amount of information transmitted. Such a system allows the operator to control both an aerial robot and a ground robot. In this work, the developed system is used for controlling multi-robots (a UAV-UGV combination) and a series of tests conducted in outdoor Finnish conditions during summer, fall, and winter 2016 and 2017.