Robotic Arm Control Using Visual Neural Networks and Hierarchical Reinforcement Learning

Reinforcement learning has been extensively explored in the field of robotic arm control, becoming a highly adaptive and flexible strategy capable of solving complex manipulation tasks. This study introduces a vision-based Hierarchical Advantage Actor-Critic algorithm for robotic arm control. For perception, we train a convolutional neural network with attention mechanism using depth images as input to output 3D coordinates of the target object, serving as observations for reinforcement learning. For decision-making, we optimize the traditional Advantage Actor-Critic algorithm with a hierarchical design of rewards, enabling the agent to quickly understand which actions are beneficial, thereby accelerating the learning process. In terms of control, we use joint position changes as the action space, which are then converted into output torques through gravity compensation. This approach allows the robot to precisely control its motion trajectory during task execution while effectively addressing the impacts of gravity and other external disturbances. We deployed our method in two tasks, achieving a success rate above 80 % in each.

I. Introduction

The field of robotics is advancing rapidly, and applying reinforcement learning (RL) methods to robotic arm control has become a major research focus in this domain. RL has achieved exciting results in several fields such as autonomous navigation [1], object grasping [2], gait control [3], human-robot collaboration [4], and group collaboration [5].