Performance Analysis and Comparison of Knowledge Distillation Among Diverse Teacher and Student Models

With the advancements in deep learning, the development of large-scale models has become essential for many machine learning tasks in various fields such as computer vision and natural language. Such large-scale models demand extensive computational power and memory, posing challenges for practical deployment on mobile and edge devices. To address these issues, knowledge distillation, a technique for reproducing the performance of large-scale models with a smaller model, has been proposed. This paper investigates the effectiveness of knowledge distillation across various combinations of teach-student model pairs for image classification to deepen the understanding of knowledge distillation and the relationship between large- and small-scale models.