FusePose: IMU-Vision Sensor Fusion in Kinematic Space for Parametric Human Pose Estimation

Commercial motion-capture systems produce excell- ent in-studio reconstructions, but offer no comparable solution for acquisition in everyday environments. We present a system for acquiring motions almost anywhere. This wearable system gathers ultrasonic time-of-flight and inertial measurements with a set of inexpensive miniature sensors worn on the garment. After recording, the information is combined using an Extended Kalman Filter to reconstruct joint configurations of a body. Experimental results show that even motions that are traditionally difficult to acquire are recorded with ease within their natural settings. Although our prototype does not reliably recover the global transformation, we show that the resulting motions are visually similar to the original ones, and that the combined acoustic and intertial system reduces the drift commonly observed in purely inertial systems. Our final results suggest that this system could become a versatile input device for a variety of augmented-reality applications.

I. Introduction

3D human pose estimation is one of the most important problems in computer vision, which is closely related to human motion analysis, action recognition, human-computer interaction and so on [1], [9], [28], [36], [55]. The common solution of this problem is to predict the 3D keypoint coordinates of a predefined human skeleton from single-view [34] or multi-view RGB images [27], [41], [47]. In this solution, it is widely known that occlusions or self-occlusion in images introduce stochastic estimation errors in extracting image features. As shown in Fig. 1, occlusion is inevitable in vision data. It may significantly reduce the performance and result in unreasonable human pose [22], [58].