A Study of Inter-Channel Knowledge Distillation Based on Inheritance and Exploration

Knowledge distillation is a hot research topic in the field of deep learning model compression. Knowledge distillation aims to transfer knowledge from a larger and optimised teacher network to a smaller learnable network and is important for improving the accuracy of student models. Existing KD methods typically consider privileged information about the similarity/consistency between the class distribution or intermediate feature representations of the teacher and student models. However directly driving the student model to simulate the features of the teacher model largely ignores the knowledge/features that are not found in the student model learning the teacher model. In this paper we propose a new inter-channel correlation framework for knowledge extraction (IE-ICKD) that divides the student model into two parts, inheritance and exploration, where the inheritance part has transferred existing learning knowledge from the teacher model to the student model by simulating the feature space of the teacher model, while the exploration part is assigned to learn the feature space not learned by the simulated teacher model. Extensive experiments have shown that our proposed model enables better performance of the student model on the CIFAR-100 and CIFAR-10 datasets, and it outperforms many existing methods. We achieved a Top-1 accuracy of 74.12% by choosing ResNet110 as the teacher and ResNet32 as the student, which is very close to the 74.31% of the teacher model, achieving distillation and demonstrating the effectiveness of our method.