Hierarchical Region-level Decoupling Knowledge Distillation for semantic segmentation

Region-level distillation can facilitate the transfer of salient region information for each channel. However, computing Kullback-Leibler divergence for the entire soft probability map can significantly limit the effectiveness of Non-target Class Knowledge Distillation. To address this issue, we propose Region-level Decoupling Knowledge Distillation. This simple and efficient approach implicitly decouples region-level distillation into Target Region Knowledge Distillation (TRKD) and Non-target Region Knowledge Distillation (NRKD), ensuring effective transfer of region-level dark knowledge present in both TRKD and NRKD. To progressively integrate global information, we further propose Hierarchical Region-level Decoupling Knowledge Distillation, which gradually aggregates global information through a simple average pooling operation, thereby facilitating the distillation of multi-scale semantic information. We conduct extensive experiments on six benchmark datasets: Cityscapes, Pascal VOC, ADE20k, and COCO Stuff164k for natural images, and Synapse and FLARE22 for medical images. The experimental and visualization results demonstrate that our proposed distillation methods achieve state-of-the-art performance in semantic segmentation tasks without introducing auxiliary modules.