Investigating Fusion Strategies on Encoder-Decoder Networks for Crop Segmentation Using SAR and Optical Image Sequences

Remote sensing imagery from different sensors enables accurate crop monitoring and mapping, supporting efficient and sustainable agricultural practices. This paper proposes a flexible multi-modal Encoder-Decoder architecture to implement and investigate different fusion strategies for crop segmentation using synthetic-aperture radar (SAR) and optical image sequences. The trained model handles each modal individually or both, allowing us to investigate scenarios where one modal is not available. Our architecture consists of two modality-specific encoders, a shared decoder, a fusion module, and three classifiers, one for each modal and one for the fusion output. Also, we propose using a partial loss function that allows training the network with scarce ground truths. The proposed approach was evaluated in a public dataset comprising seven multitemporal SAR images and four multitemporal optical images from a tropical agricultural region in Brazil. We report results for feature and decision fusion strategies and discuss the benefits of using each of them for multi-modal crop segmentation.