Spatial-Temporal Consistency Refinement Network for Dynamic Point Cloud Frame Interpolation

Point cloud frame interpolation aims to improve the frame rate of a point cloud sequence by synthesising intermediate frames between consecutive frames. Most of the existing works only use the scene flow or features, not fully exploring their local geometry context or temporal correlation, which results in inaccurate local structural details or motion estimation. In this paper, we organically combine scene flows and features to propose a two-stage network based on residual-learning, which can generate spatially and temporally consistent interpolated frames. At the Stage 1, we propose the spatial-temporal warping module to effectively integrate multi-scale local and global spatial features and temporal correlation into a fusion feature, and then transform it into a coarse interpolated frame. At the Stage 2, we introduce the residual-learning structure to conduct spatial-temporal consistency refinement. A temporal-aware feature aggregation module is proposed, which can facilitate the network adaptively adjusting the contributions of spatial features from input frames, and predict the point-wise offset as the compensations due to coarse estimation errors. The experimental results demonstrate our method achieves the state-of-the-art performance on most benchmarks with various interpolated modes. Code is available at https://github.com/renlancao/SR-Net.

I. Introduction

With rapid developments in 3D acquisition equipment, point cloud has become one of the most widely-used 3D digital representation for real-world persons, objects or scenes, which can be encountered in many applications, such as AR/VR/XR, urban digital twins, Metaverse, etc. However, limited by sensor performance and high cost of acquisition, the frame rate of captured dynamic point clouds (i.e., a sequence, formed by consecutive frames along the time) is typically low (about 25 Hz), which leads to poor temporal consistency. It is known that higher-frame-rate sequences could bring more immersive visual experience for users. Therefore, point cloud frame interpolation, which can increase the frame rate of a point cloud sequence by generating the intermediate frames between consecutive frames, is needed. Fig. 1.

Visualization of interpolated results on the swing.