Multiview Fusion Driven 3-D Point Cloud Semantic Segmentation Based on Hierarchical Transformer

Three-dimensional semantic segmentation is a key task of environment understanding in various outdoor scenes. Due to the sparsity and varying density of point clouds, it becomes challenging to obtain fine-gained segmentation results. Previous point-based and voxel-based methods suffer from the expensive computational cost. Recent 2-D projection-based methods, including range-view (RV), bird-eye-view (BEV), and multiview fusion methods, can run in real time, but the information loss during the projection leads to the low accuracy. Also, we find that the occlusion and interlacing problems exist in single projection-based methods and most multiview fusion networks only focus on the output-level fusion. Considering the above issues, we propose a multilevel multiview fusion network using attention modules and hierarchical transformer, which ensures the effectiveness and efficiency mainly by the following three aspects: 1) the spatial-channel attention module (SCAM) integrates contextual information between points and learn differences of each channel’s features; 2) the proposed geometry-based multiprojection fusion module (GMFM) achieves the geometric feature alignment between RV and BEV and fuses the features of the two views at both feature level and output level; and 3) we introduce KPConv to replace KNN, which can reduce the information loss during the postprocessing. Experiments are conducted on both structured and unstructured datasets, including urban dataset SemanticKITTI and off-road dataset Rellis3D. Our results achieve a better performance compared to other projection-based methods and are comparable with the state-of-the-art Cylinder3D.