Evolving Ensemble Model based on Hilbert Schmidt Independence Criterion for task-free continual learning

Continual Learning (CL) aims to extend the abilities of deep learning models for continuously acquiring new knowledge without forgetting. However, most CL studies assume that task identities and boundaries are known, which is not a realistic assumption in areal scenario. In this work, we address amore challenging and realistic situation in CL, namely the Task-Free Continuous Learning (TFCL), where an ensemble of experts is trained on non-stationary data streams without having any task labels. To deal with TFCL, we introduce the Evolving Ensemble Model (EEM), which can dynamically build new experts into a mixture, thus adapting to the changing data distributions while continuously learning new data sets. To ensure a compact network architecture for EEM during training, we propose a novel expansion mechanism that considers the Hilbert- Schmidt Independence Criterion (HSIC) for evaluating the statistical consistency between the knowledge learned by each expert and that corresponding to the given data. This expansion mechanism does not require storing all previous samples and is more efficient as it performs statistical evaluations in a low-dimensional feature space inferred by a deep network. We also propose anew dropout mechanism for selectively removing unimportant stored samples from the memory buffer used for storing the continuously incoming data before they are used for training. The proposed dropout mechanism ensures the diversity of information being learnt by the experts of our model. We perform extensive TFCL tests which show that the proposed approach achieves the state of the art. The source code is available at https://github.com/dtuzi123/HSCI-DEM.