Global Adaptive Second-Order Transformer for Remote Sensing Image Semantic Segmentation

In the domain of remote sensing (RS) image analysis, capturing global context is the key for precise semantic segmentation. Current vision transformer (ViT) advance this field by addressing convolutional neural network's (CNN) local receptive field limitations. However, ViT predominantly rely on the first-order information in image to establish global relationships, often overlooking the potential of second-order information, which is crucial for enhancing the discrimination of ground objects that exhibit high similarity and constant changes. To address this issue, we propose a global adaptive second-order transformer network (GASOT-Net). Specifically, the proposed global adaptive second-order transformer (GASOT) enhances the existing ViT structure by mining second-order information and adaptively fusing it with the first-order information during the process of establishing global dependency relationships. This approach enables the extraction of more discriminative features, thereby enriching the representation of global features. In addition, the local feature aggregation module (LFAM) is proposed to effectively aggregate features from different stages of CNN as input to the GASOT blocks. Moreover, to refine boundaries of complex ground objects, the global feature enhancement module (GFEM) is used in the decoder stage. In particular, GFEM includes two sub modules-feature shift module (FSM) and hierarchical feature fusion module (HFFM). FSM is used to enhance the local feature representation at first, and then, HFFM hierarchically aggregates local and global features from different stages. We conduct extensive experiments on four benchmark RS datasets, and the results show that our GASOT-Net outperforms other state-of-the-art methods. The code will be available at: https://github.com/j136812832/GASOT-Net.