Visual Perspective Taking for Opponent Behavior Modeling

In order to engage in complex social interaction, humans learn at a young age to infer what others see and cannot see from a different point-of-view, and learn to predict others’ plans and behaviors. These abilities have been mostly lacking in robots, sometimes making them appear awkward and socially inept. Here we propose an end-to-end long-term visual prediction framework for robots to begin to acquire both these critical cognitive skills, known as Visual Perspective Taking (VPT) and Theory of Behavior (TOB). We demonstrate our approach in the context of visual hide-and-seek – a game that represents a cognitive milestone in human development. Unlike traditional visual predictive model that generates new frames from immediate past frames, our agent can directly predict to multiple future timestamps (25 s), extrapolating by 175% beyond the training horizon. We suggest that visual behavior modeling and perspective taking skills will play a critical role in the ability of physical robots to fully integrate into real-world multi-agent activities.

I. INTRODUCTION

Imagine that you see your colleague trying to open the door, only to find that the door is blocked by the doorstop hidden on the other side. Without exchanging words, you rush to remove the doorstop, allowing the door to swing open. You took this action because you understood what your colleague was trying to do (open the door), and you understood that she could not see the doorstop (from her viewpoint). We refer to these intuitive abilities as Behavior Modeling and Visual Perspective Taking respectively. Visual Perspective Taking [1]–[6] is essential for many useful multiagent skills such as understanding social dynamics and relationships, engaging in social interactions, and understanding intentions of others. A simple action such as getting out of the way of a busy person, or assisting a person about to sit down, involve both of these abilities.