Ego+X: An Egocentric Vision System for Global 3D Human Pose Estimation and Social Interaction Characterization

Egocentric vision is an emerging topic, which has demonstrated great potential in assistive healthcare scenarios, ranging from human-centric behavior analysis to personal social assistance. Within this field, due to the heterogeneity of visual perception from first-person views, egocentric pose estimation is one of the most significant prerequisites for enabling various downstream applications. However, existing methods for egocentric pose estimation mainly focus on predicting the pose represented in the camera coordinates from a single image, which ignores the latent cues in the temporal domain and results in less accuracy. In this paper, we propose Ego+X, an egocentric vision based system for 3D canonical pose estimation and human-centric social interaction characterization. Our system is composed of two head-mounted egocentric cameras, where one is faced downwards and the other looks outwards. By leveraging the global context provided by visual SLAM, we first propose Ego-Glo for spatial-accurate and temporal-consistent egocentric 3D pose estimation in the canonical coordinate system. With the help of an egocentric camera looking outwards, we then propose Ego-Soc by extending Ego-Glo to various social interaction tasks, e.g., object detection and human-human interaction. Quantitative and qualitative experiments have been conducted to demonstrate the effectiveness of our proposed Ego+X.

I. Introduction

Egocentric vision can offer sufficient information about how people perceive the world and interact with the environment from a human-centric perspective, furnishing diverse opportunities for the analysis of human behavior and cognition [1], [2]. In general, an egocentric vision system can be composed of either head-mounted or chest-mounted cameras, capturing the visual data in a free-living environment with increased mobility and flexibility. Recent progress in egocentric vision has been widely penetrated in daily life assistive healthcare, including human behavior analysis [2], human-machine interaction [3], and social interaction characterization and assistance [4].