CNN-GCN-based dual-stream network for scene classification of remote sensing images

Scene classification is an important research topic, which aims at assigning a semantic label to a given image. High-Spatial-Resolution (HSR) images contain abundant information of ground objects, such as geometric structure and spatial layout. Complex HSR images are difficult to interpret effectively. Extracting discriminative features is the key step to improve classification accuracy. Various methods for constructing discriminative representations, including handcrafted feature-based methods and deep learning-based methods, have been proposed. The former methods focus on designing different handcrafted features via professional knowledge and describing a scene through single feature or multifeature fusion. However, for a complex scene, handcrafted features show limited discriminative and generalization capabilities. Deep learning-based methods, due to the powerful capability of feature extraction, have made incredible progress in the field of scene classification. Compared with the former methods, Convolutional Neural Networks (CNNs) can automatically extract deep features from massive HSR images. Nevertheless, CNNs merely focus on global information, which makes it fail to explore the context relationship of HSR images. Recently, Graph Convolutional Networks (GCNs) have become an important branch of deep learning, and they have been adopted to model spatial relations hidden in HSR images via graph structure. In this paper, a novel architecture termed CNN-GCN-based Dual-Stream Network (CGDSN) is proposed for scene classification. The CGDSN method contains two modules: CNN and GCN streams. For the CNN stream, the pretrained DenseNet-121 is employed as the backbone to extract the global features of HSR images. In the GCN stream, VGGNet-16 that is pretrained well on ImageNet is introduced to generate feature maps of the last convolutional layer. Then, an average pooling is organized for downsampling before the construction of an adjacency matrix. Given that every image is represented by a graph, a GCN model is developed to demonstrate context relationships. Two graph convolutional layers of the GCN stream are followed by a global average pooling layer and a Fully Connected (FC) layer to form the context features of HSR images. Lastly, to fuse global and context features adequately, a weighted concatenation layer is constructed to integrate them, and an FC layer is introduced to predict scene categories. The AID, RSSCN7, and NWPU-RESISC45 data sets are chosen to verify the effectiveness of the CGDSN method. Experimental results illustrate that the proposed CGDSN algorithm outperforms some state-of-the-art methods in terms of Overall Accuracies (OAs). On the AID data set, the OAs reach 95.62% and 97.14% under the training ratios of 20% and 50%, respectively. On the RSSCN7 data set, the classification result obtained by the CGDSN method is 95.46% with 50% training samples. For the NWPU-RESISC45 data set, the classification accuracies achieved via the CGDSN method are 91.86% and 94.12% under the training ratios of 10% and 20%, respectively. The proposed CGDSN method can extract discriminative features and achieve competitive accuracies for scene classification. © 2021, Science Press. All right reserved.