EXPANSE: A Continual and Progressive Learning System for Deep Transfer Learning

Deep transfer learning (DTL) techniques attempt to tackle the limitations of deep learning, the dependency on extensive training data and the training costs by reusing obtained knowledge from source data for target data. However, the current DTL techniques suffer from either catastrophic forgetting dilemma (losing the previously obtained knowledge) or overly biased pre-trained models (harder to adapt to target data) in fine-tuning pre-trained models or freezing a part of the pre-trained model, respectively. We propose a new continual/progressive learning approach for deep transfer learning to tackle these limitations. We extend the pre-trained model by expanding pre-trained layers (adding new nodes to each layer) in the model instead of only adding new layers. Hence the method is named EXPANSE. Our experimental results confirm that we can tackle distant source and target data using this technique. At the same time, the final model is still valid on the source data, achieving a promising deep continual learning approach. Moreover, we offer a new way of training deep learning models inspired by the human education system. We termed this two-step training: learning basics first, then adding complexities and uncertainties. The evaluation implies that the two-step training extracts more meaningful features and a finer basin on the error surface since it can achieve better accuracy compared to regular training.