Building and road detection from remote sensing images based on weights adaptive multi-teacher collaborative distillation using a fused knowledge

Knowledge distillation is one effective approach to compress deep learning models. However, the current distillation methods are relatively monotonous. There are still rare studies about the combination of distillation strategies using multiple types of knowledge and employing multiple teacher models. Besides, how to optimize the weights among different teacher models is still an open problem. To address these issues, this paper proposes a novel approach for knowledge distillation, which effectively enhances the robustness of the distilled student model by a weights adaptive multi-teacher collaborative distillation. Moreover, the proposed method utilizes feature knowledge exchange guidance between teacher networks to transfer more comprehensive feature knowledge to the student model, which further improves the learning capability of hidden layers' details. The extensive experimental results demonstrate that the proposed method achieves state-of-the-art performance on Massachusetts Roads Dataset, LRSNY Roads Dataset, and WHU Building Dataset. Specifically, under the guidance of the first ensemble of teacher networks, we obtained IoU scores of 47.33%, 78.15%, and 80.71%, respectively. Under the guidance of the second ensemble of teacher networks, we obtained IoU scores of 48.56%, 79.51%, and 81.35%, respectively.