TwinFusion: High Framerate Non-rigid Fusion through Fast Correspondence Tracking

Real time non-rigid reconstruction pipelines are extremely computationally expensive and easily saturate the highest end GPUs currently available. This requires careful strategic choices to be made about a set of highly interconnected parameters that divide up the limited compute. At the same time, offline systems, prove the value of increasing voxel resolution, more iterations, and higher frame rates. To this end, we demonstrate a set of remarkably simple but effective modifications to these algorithms that significantly reduce the average per-frame computation cost allowing these parameters to be increased. Specifically, we divide the depth stream into sub-frames and fusion-frames, disabling both model accumulation (fusion) and non-rigid alignment (model tracking) on the former. Instead, we efficiently track point correspondences across neighboring sub-frames. We then leverage these correspondences to initialize the standard non-rigid alignment to a fusion-frame where data can then be accumulated into the model. As a result, compute resources in the modified non-rigid reconstruction pipeline can be immediately re-purposed to increase voxel resolution, use more iterations or to increase the frame rate. To demonstrate the latter, we leverage recent high frame rate depth algorithms to build a novel “twin” sensor consisting of a low-res/highfps sub-frame camera and a second low-fps/high-res fusion camera.

1. Introduction

The use of cameras to digitize the geometry, texture, lighting and motion of arbitrary scenes is a fundamental problem in computer vision. General monocular solutions remain elusive, but practical algorithms have been developed that leverage motion, shape or appearance priors, and/or require instrumentation of the scene using motion markers or multiple calibrated cameras. Figure 1.

(Top) our TwinPod hybrid depth camera captures highspeed performance with a pair of high (resp. Low) framerate and low (resp. high) resolution sensors. (bottom-left) The reconstruction in the canonical frame obtained by our re-implementation of DynamicFusion on the 30FPS stream. (bottom-right) Our TwinFusion algorithm efficiently resolves motion from the high-framerate camera and exploits this information to guide the high-resolution non-rigid reconstruction.