Convolution Meets Transformer: Efficient Hybrid Transformer for Semantic Segmentation with Very High Resolution Imagery

In this paper, we introduce an efficient and lightweight hybrid Transformer architecture, ingeniously integrating convolutions within Transformer blocks for semantic segmentation of remote sensing Very High Resolution (VHR) imagery. To simultaneously avoid the high computational complexity in the shallow layers and capture the local representations of the VHR images, we propose the Group-Team Convolution Modulation (GTCM) module that uses convolutions to approximate the effect of attention mechanisms and modulates features in channel dimension. Additionally, to enlarge the effective receptive field (ERF) in the decoder, based on the grouping philosophy, we adopt dilated convolutions with multiple dilated rates to further enhance the performance. The superiority and efficiency of our proposed hybrid structure are demonstrated by outperforming state-of-the-art methods on the Vaihingen and Potsdam datasets with relatively lower complexity and fewer parameters.

1. INTRODUCTION

The rapid development of spectral and optical sensors for satellites has increasingly provided large volumes of earth observation data that meets the data-rich prerequisite for developing automatic analyzing algorithms, especially deep learning (DL)-based methods. Among these, Vision Transformer (ViT) DL models have gained considerable traction in the Remote Sensing (RS) domain [1] and achieved superior performance in various RS applications, such as RS semantic segmentation on Very High Resolution (VHR) [2] and multispectral images [3]. Apart from achieving more uniform global representations compared to Convolutional Neural Networks (CNNs), ViTs can also scale effectively to accommodate large-scale data and model sizes, so they are regarded as the foundation vision models for large-scale RS models [4]. Despite their advantages, the substantial computational complexity introduced by attention mechanisms in Transformer architectures undermines the performance benefits of the powerful models on resource-constrained devices, particularly with very high-resolution inputs [5].