Generative Dataset Distillation

As the structure and function of deep learning models become more and more complex, huge amounts of data are often required for training to avoid over-fitting. Dataset distillation is a method for reducing dataset sizes by learning a small amount of synthetic samples which take almost the same training effect as a larger dataset, and the method behaves as transferring the information of the original dataset to a smaller one. This has several benefits like accelerating model training, saving energy, and reducing storage space. However, the distilled data of current methods of dataset distillation is highly dependent on the training model and looks very noisy. And no metric has been proposed yet to reveal the generalization quality of the distilled data. In this paper, we discuss the generalization of the distillation and propose an approach to distill with auxiliary generative neural networks to enhance the performance. For example, training the LeNet with 10 distilled images (one per class) and random initialization results in over 82% test accuracy on MNIST, and the distilled samples look much more informative with better generalization. Besides, we demonstrate an application in key feature identification of images with our method, where the features for distinction of two classes of data samples are distilled.