Sequential Point Cloud Upsampling by Exploiting Multi-Scale Temporal Dependency

In this work, we propose a new sequential point cloud upsampling method called SPU, which aims to upsample sparse, non-uniform, and orderless point cloud sequences by effectively exploiting rich and complementary temporal dependency from multiple inputs. Specifically, these inputs include a set of multi-scale short-term features from the 3D points in three consecutive frames (i.e., the previous/current/subsequent frame) and a long-term latent representation accumulated throughout the point cloud sequence. Considering that these temporal clues are not well aligned in the coordinate space, we propose a new temporal alignment module (TAM) based on the cross-attention mechanism to transform each individual feature into the feature space of the current frame. We also propose a new gating mechanism to learn the optimal weights for these transformed features, based on which the transformed features can be effectively aggregated as the final fused feature. The fused feature can be readily fed into the existing single frame-based point cloud upsampling methods (e.g., PU-Net, MPU and PU-GAN) to generate the dense point cloud for the current frame. Comprehensive experiments on three benchmark datasets DYNA, COMA, and MSR Action3D demonstrate the effectiveness of our method for upsampling point cloud sequences.

I. Introduction

To better understand 3D shapes, objects, and scenes captured by LIDAR sensors and RGB-D cameras, rapidly increasing attention has been paid to 3D point cloud processing algorithms in the community of computer vision, robotics, and mixed realities. Significant progress has been made in recent works, which mainly focus on high-level tasks, such as 3D object classification, segmentation, and detection [1]–[9]. However, most of these works still face considerable challenges since the point clouds captured in the real scenarios are usually noisy, sparse and non-uniform.