JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups

Understanding human social behaviour is crucial in Computer vision and robotics. Micro-level observations like in-dividual actions fall short, necessitating a comprehensive approach that considers individual behaviour, intra-group dynamics, and social group levels for a thorough under-standing. To address dataset limitations, this paper intro-duces JRDB-Social, an extension of JRDB [2]. Designed to fill gaps in human understanding across diverse indoor and outdoor social contexts, JRDB-Social provides annotations at three levels: individual attributes, intra-group in-teractions, and social group context. This dataset aims to enhance our grasp of human social dynamics for robotic applications. Utilizing the recent cutting-edge multi-modal large language models, we evaluated our benchmark to ex-plore their capacity to decipher social human behaviour.

1. Introduction

Human social behaviour understanding finds numerous Applications in computer vision and robotics. Simply observing the micro-level information like the actions of an indi-vidual is inadequate for a comprehensive understanding of human behaviour because humans are inherently social beings and require analysis within a broader social context. Therefore, a comprehensive and multi-layered approach is required to perceive human social behaviour thoroughly. For example, in security and surveillance systems, integrating individual-level data, identifying social groups, and taking context into account significantly enhance the overall capacity to better understand crowd behaviors [3]. Additionally’ this integration fosters more natural and intuitive experiences in human-robot interaction like telerobots [4], coworker robots [5] and social robots [6].