Robot-centric Activity Recognition from First-Person RGB-D Videos

We present a framework and algorithm to analyze first person RGBD videos captured from the robot while physically interacting with humans. Specifically, we explore reactions and interactions of persons facing a mobile robot from a robot centric view. This new perspective offers social awareness to the robots, enabling interesting applications. As far as we know, there is no public 3D dataset for this problem. Therefore, we record two multi-modal first-person RGBD datasets that reflect the setting we are analyzing. We use a humanoid and a non-humanoid robot equipped with a Kinect. Notably, the videos contain a high percentage of ego-motion due to the robot self-exploration as well as its reactions to the persons' interactions. We show that separating the descriptors extracted from ego-motion and independent motion areas, and using them both, allows us to achieve superior recognition results. Experiments show that our algorithm recognizes the activities effectively and outperforms other state-of-the-art methods on related tasks.

1. Introduction

We live in an era where machines, and in particular robots, are starting to become part of humans' lives. We would like them to help us, to learn from us, to be our companion, to detect threats. In order to do so, it is necessary that robots start to understand humans' emotions, humans' intentions and humans' behaviors that are directly related to them. A child can show affection hugging a robot. A person could try to damage a surveillance robot throwing something at it. The problem we consider regards the recognition of those activities performed by humans that are explicitly associated to the presence or behavior of the robot. We refer to this new problem as robot-centric activity recognition.