Deep State Space Visual Prediction for Robotic Manipulation

We propose a deep state space visual prediction framework for robotic manipulation. A circumstance is concerned when part of the robotic arm is clipped out of the image captured by the camera hung above. A two-stage Deep Neural Networks (DNNs) architecture is suggested to perform state extracting and image restoring, and a Sparse Bayesian Learning (SBL) approach is applied to identify the State Space equation representing the dynamics between the state generated by the former Networks and the position of the robotic arm’s end-effector. The proposed framework is able to predict the future robotic arm’s location in a image-based fashion by incorporating the current robotics image and future trajectory of the end-effector’s position in a large prediction time period when most part of the robotic arm is unseen by the camera. The experiments operated on UR5 (6-DOF robotic arm) show a satisfying prediction results measured by IoU (Intersection over Union) of 0.947 in a robotic arm manipulation which predicts from the start position to the end position.