Continuous Image Generation From Low-Update-Rate Images and Physical Sensors Through a Conditional GAN for Robot Teleoperation

When a robot is teleoperated, its operator control is based on transmitted images. Network limitations and/or a remote distance usually cause delays or interruptions of the image transmission, which is one of the reasons for the instability of teleoperation systems. In this article, we propose a high-update-rate image generation method using past low update image and current grip position and electrical motor current of gripper received by sensors during teleoperation via a conditional generative adversarial network. The main challenge is that such a network can generate current high-update-rate images from past low-update-rate one, the current high-update-rate grip force, and the grip angle. We equipped a robot gripper with a camera and a grip force sensor and collected a large data set of robot vision, grip force, and grip angle sequences; objects with deformation, including irregular deformation, and rigid objects were tested in the experiment to verify the possibility of high-update-rate image generation under various grip conditions. We found that the proposed network allows the generation of current images with high update rate.

I. Introduction

Over the past few decades, various teleoperation systems have been developed for remote robot control. They have been used for a variety of tasks, such as operating robotic arms or mechanical systems in remote areas (e.g., for controlling surgical robots, and handling harmful or toxic materials like in nuclear reactors). A teleoperation system consists of a master system for manipulation by an operator and a slave robot remotely controlled by the master movements. In a bilateral teleoperation system, the master and the slave are connected through communication networks that transfer various information, including the master and slave positions, the interaction forces between master system and operator and between slave system and environment, and visual data. Several transparency and stability studies have been recently conducted to understand how to obtain haptic feedback [1]–[4]. In teleoperation, the operator needs to effectively manipulate the slave robot through tactile and kinesthetic feedback; however, visual feedback is still one of the most important factors when a person manipulates a robot remotely. Since a human being obtains more than 90% of information through vision [5], trouble may arise if the operator cannot rely on accurate visual data.