Multimodal Remote Sensing Data Classification Based on Gaussian Mixture Variational Dynamic Fusion Network

With the development of sensor technology, the rational use of multimodal data has become a research hotspot in the field of remote sensing. The multimodal fusion method can effectively improve the accuracy of remote sensing data classification by using the complementary information of different modalities. However, the existing multimodal fusion methods face many challenges, including difficulties in suppressing spectral noise, fully mining contextual information, and learning the strong adaptive fusion pattern. To address the above challenges, a Gaussian mixture variational dynamic fusion network (GM-VDFN) is proposed. First, a multimodal multiscale spatial graph is constructed, and the graph convolution is used to learn the multiscale features. In this process, a spatial topology constraint based on GM (STC-GM) is proposed, which suppresses spectral noise by constraining the topological consistency of the two modalities. Second, a multiscale dynamic graph aggregation module (MDGAM) is constructed, which can capture the shareable class identification information from multiscale features and mine personalized fusion patterns suitable for each sample. Finally, the evidence lower bound for the multimodal joint distribution is derived, and a multimodal variational autoencoder (M-VAE) is designed. Optimizing the evidence lower bound to model multimodal joint distributions, thereby learning the strong adaptive fusion pattern between modalities. Experimental results on four fusion datasets (Houston 2013, Trento, MUUFL, and Houston 2018) show that GM-VDFN achieved state-of-the-art performance in multimodal remote sensing data classification tasks.