MASNet: Road Semantic Segmentation Based on Multiscale Modality Fusion Perception

With the growing complexity of driving environments, relying solely on a single sensor for scene understanding is no longer sufficient. To address this issue, this article introduces a novel dual-modal semantic segmentation network called multiscale adaptive stream network (MASNet) for road environments, which leverages optical image features to complement the LiDAR point cloud features. To achieve feature uniformity, we project 3-D point clouds onto the image coordinate system and utilize an image information mask matrix to filter out effective 2-D point clouds. In the backbone network of the LiDAR and image branches, SalsaNext and ResNet34 are employed for parallel feature encoding, respectively. Moreover, a module based on residual mapping and point attention mechanism is introduced to unidirectionally transfer image features to the LiDAR branch across multiple encoding stages. The multiexpansion rate multicascaded atrous spatial convolutional pooling pyramid (MmASPP) structure is designed between the encoder and decoder of the LiDAR branch, comprising parallel bottleneck structures with different dilation rates, effectively capturing multiscale contextual information, enhancing pixel correlations, and achieving dense pixel feature aggregation. Additionally, we propose an adaptive synergy difference loss function based on cross-entropy, which measures semantic information discrepancies during cross-modal operations by interacting and propagating predicted label distributions at the network output layer. The modal computation weights in the loss function are assigned based on modal differences to highlight the perceptual advantages of specific modalities in application scenarios. Compared to several popular networks, MASNet demonstrates superior segmentation performance, and the design of the MmASPP parallel bottleneck structure reduces the parameter count in context modules while achieving higher accuracy.