Incremental Deep Learning Method for Object Detection Model Based on Knowledge Distillation

With the advent of the Internet of Everything era, the number of IoT devices with object detection capability has exploded. Accordingly, massive amounts of real-time data are generated at the edges of the network. Thus, edge computing has become an emerging computing paradigm that has the characteristics of low latency, low bandwidth and high security. While the traditional deep learning approaches usually assume that all data have been obtained before model training, a large number of new data and new categories are often gradually generated and obtained over time in real edge computing environment. In order to execute the object detection task efficiently on resource-constrained edge devices when the input data samples are accumulated and updated in batches, an incremental learning method based on knowledge distillation of multiple intermediate layers (ILMIL) was proposed in this paper. First, to preserve the obtained knowledge from existing data, a new metric called MFRRK was proposed for covering knowledge from multiple intermediate network layers. The discrepancy between the intermediate layers’ features of teacher model and student model were added in ILMIL to model training. Compared to the recent incremental learning methods based on knowledge distillation, the student model adapted ILMIL was able to alleviate forgetting by learning more knowledge of the old classes from the intermediate layers of the teacher model. Then, the incremental training for current model was conducted to avoid resource costs of using multiple independent models for training. To further reduce the model complexity, the model pruning technique can be used before knowledge distillation to compress the current model. Extensive experiments under different scenarios and conditions demonstrated that the proposed ILMIL method can effectively reduce the model calculation and storage costs, alleviate the catastrophic forgetting of existing knowledge, and maintain acceptable inference accuracy. © 2022 Editorial Department of Journal of Sichuan University. All rights reserved.