Learning a dual-branch classifier for class incremental learning

Catastrophic forgetting is a non-trivial challenge for class incremental learning, which is caused by new knowledge learning and data imbalance between old and new classes. To alleviate this challenge, we propose a class incremental learning method with dual-branch classifier. First, inspired by ensemble learning, the proposed method constructs a dual network consisting of two complementary branches to alleviate the impact of data imbalance. Second, activation transfer loss is employed to reduce the catastrophic forgetting from the view of feature representation, preserving the feature separability of old classes. Third, we use the nearest class mean classifier with natural advantages for classification. Moreover, we formulate an end-to-end training algorithm for the feature extraction and classifier, to boost module matching degree. Extensive evaluation results show our proposed method achieves nice incremental recognition ability with less training time. Moreover, the ablation study shows the importance and necessity of dual-branch structure, end-to-end training, and activation transfer loss.