Graph-Based Representation Knowledge Distillation for Image Classification

The core idea of knowledge distillation is to use a large model as the teacher network to guide a small model as the student network, improving the performance of the student network in image classification tasks. Existing knowledge distillation methods often extract category probability or feature information as knowledge from a single input sample. They could not model the relationships between samples, decreasing the network’s representation learning ability. To solve this problem, this paper introduces a graph convolutional neural network, which treats the input sample set as graph nodes to construct a relationship graph. Each sample in the graph could aggregate information from other samples, improving its own representation ability. This paper constructs the distillation loss of graph representation knowledge from the perspectives of graph nodes and relationships. It uses meta-learning to guide the student network to adaptively learn better graph representations from a teacher network, thereby improving the graph modeling ability of the student network. Compared to the baseline method, the graph-based representation knowledge distillation method improves the classification accuracy by 3.70% on the 100-classification dataset published by Canadian Institute For Advanced Research. The result indicates that the proposed method makes the student network learn a more discriminative feature space, thereby improving its image classification ability. © 2024 Chinese Institute of Electronics. All rights reserved.