Robot Manipulation of Dynamic Object with Vision-Based Reinforcement Learning

With the advanced development of AI, Deep Reinforcement Learning (DRL) is one way to let robots learn how to complete tasks independently. Based on DRL algorithms, robot manipulation in the tabletop scene is regarded as a typical environment for testing and validation. Within this context, we designed a vision-based scene to explore the performance of DRL algorithms in manipulating a dynamic object. Contrary to previous work with simple moving trajectories, the complexity of our approach was increased by applying a randomized vibrating trajectory. To better capture visual cues in motion, we also incorporated a Long Short-term Memory (LSTM) module to process the temporal information between the serial frames. By comparing with setups that only used basic CNN encoders, we found the addition of LSTM can drastically increase the learning efficiency under model-free RL algorithms like SAC. The learning inflection point of applying LSTM appeared at 300k time steps while the others were delayed to 500k time steps or even later, which underscores the necessity of extracting temporal features in dynamic scenes. Furthermore, we observed that by integrating LSTM, there is no need to explicitly append kinematic information, which also supports the idea that temporal information inherently encodes kinematics data.