QMKD: A Two-Stage Approach to Enhance Multi-Teacher Knowledge Distillation

Knowledge distillation serves as an effective model compression technique, enhancing the performance of a smaller model through the transfer of knowledge. Previous research on multi-teacher knowledge distillation has predominantly focused on exploring feature correlations between each teacher model and the student model to improve the student's performance. However, these methods often overlook the potential information embedded in the soft labels generated by teachers. This paper delves into multi-teacher knowledge distillation techniques by leveraging soft-label information to uncover latent associations between multiple teachers and a single student. This paper introduces a novel method called QMKD, which utilizes reinforcement learning to extract latent information from both teacher soft labels and student representations, thereby enhancing the knowledge distillation process. The proposed approach divides the knowledge distillation into two stages: NAKD and RAKD. During training, NAKD is initially employed to facilitate knowledge transfer from multiple teachers to the student. Subsequently, RAKD is applied to further explore the latent associations between multiple teachers and the student. Experimental results demonstrate that the proposed method exhibits superior performance on classical datasets. Additionally, it consumes fewer computational resources compared to traditional feature-based multi-teacher knowledge distillation methods.