Ensemble-based Knowledge Distillation for Semantic Segmentation in Autonomous Driving

We investigate the potential of U-Net in performing knowledge distillation, both in the posture of Teacher and Student network. The experiments evaluate multiple configurations of the distillation procedure, as well as various Teacher-Student pairs, in order to exploit the capabilities of the architecture, as it is rarely used in this situation. The study was performed by applying the standard distillation process (standard KD), a pixel-wise approach of transferring Teacher predictions to the Student, as well as an Ensemble Teacher method, whose mean prediction is utilized in the training process of the student. These methods improved the results of the Student models to an extent worthy of being taken into consideration.