Oriented Pedestrian Social Interaction Modeling and Inference

In order to drive and operate safely around humans, future autonomous vehicles will be expected to perceive visual scenes and predict human behaviors beyond explicit visual features. Inferring human interactions, for example, plays an indispensable role in predicting pedestrian trajectories, because social actions such as walking together, gathering, holding hands, and talking, influence where and how people move relative to each other and their environment. Existing methods for semantic action recognition and labeling provide inputs that, while useful to human operators, cannot be used to improve predictions made by autonomous vehicles. This paper presents a graphical model approach for jointly inferring pedestrian interactions from short video clips over time. New Markov random field algorithms are presented for modeling social interactions probabilistically using spatial and temporal observations obtained over short video clips, at a time scale useful for making real-time decisions such as collision avoidance. Experiments conducted using real-world pedestrian streaming videos show that the average interaction-inference accuracy of the proposed approach is approximately 94.6%.