Online Knowledge Distillation via Collaborative Learning with Enhanced Diversity and Gradual Ensemble

In general, knowledge distillation transfers knowledge from a large pre-trained teacher network to a small student network. This is usually referred to as offline distillation and requires two stages of learning: learning of the teacher network and that of the student network. In contrast, in online distillation, the outputs of multiple student networks are integrated and regarded as the output of the teacher network. Compared to offline distillation, this method has the advantage that it does not require a large pre-trained teacher network and can be trained online. However, since online distillation uses ensemble learning, it is extremely important to consider the following two aspects. First, it is important to maintain the diversity of individual student networks during learning. Second, since the performance of each student network is poor at the beginning of the learning and improves as the learning progresses, the distillation needs to account for this change in performance. In this paper, we propose a novel online knowledge distillation method that takes these two points into account: the diversity of individual student networks is enhanced by providing different training data to each student network, and the degree of the ensemble learning is gradually increased by introducing a rump-up function into the loss function. Experiments using CIFAR-100 and Tiny ImageNet show that the proposed method can improve the generalization ability.