Online Knowledge Distillation Based on Multi-stage Multi-generative Adversarial Network

Knowledge distillation is one of the model compression techniques, where the key challenge is how to extract rich and general knowledge from teacher models, thus reducing the performance gap between the student models and the teacher models. The feature representation of the strong convergence of the teacher models has a heavy constraint on the student models. which brings certain influence on the quality of the student models. This paper designs an online knowledge distillation based on multi-stage multi-generative adversarial network (MS-MGAN), which enables the student model and the teacher model to learn together in multiple stages. First, the teacher model training is divided into several stages, and the training results of each stage are used to guide the student model training. Second, the student model is based on the convolutional block for layer-wise greedy training. Then, the generative adversarial network is integrated into the multi-stage training, with each convolutional block of the student model as the generator and the corresponding block of the teacher model as the discriminator, so that the student model can better follow and even surpass the performance of the teacher model. As demonstrated on several benchmarks, the proposed method can achieves surprising satisfied results, even outperforming the teacher model on some tasks.