Fusing exocentric and egocentric real-time reconstructions for embodied immersive experiences

Experiencing one’s own body in virtual and mixed reality can enhance applications such as 3D teleconferencing, physical and psychological rehabilitation, and natural 3D user interfaces. Embodied experiences require a dynamic virtual body to represent the user. Typical virtual bodies consist of rigged mesh models which are animated using expensive and cumbersome motion capture systems, or heavy reliance on models of human movement. Models of human appearance and movement are notoriously susceptible to undesirable and "uncanny" appearances, and are often unconvincing as a result. An alternative is to reconstruct the user in real time without relying on a motion capture system or on visual or movement models. With this approach the appearance of the virtual body and its motion are inherently natural by virtue of being directly captured, and embodied experiences are possible even at low levels of detail. However, the necessarily sparse arrangement of reconstruction cameras often produces incomplete virtual bodies. This is partly due to the disparity between the egocentric (first-person) view of the virtual body and the typically exocentric (third-person) perspectives of the reconstruction cameras. In this paper we present a method for reconstructing a more complete view of the user with a minimal number of cameras by combining head-worn egocentric with exocentric depth-sensing cameras, with a focus on the first-person view of the virtual body. We describe our approach to producing the virtual body, including camera registration methods and key technical performance metrics. We also provide insights from a user study with 26 participants indicating that our approach has the potential to increase the sense of embodiment and the perception of the completeness of the virtual body.