GAN Latent Space Manipulation Based Augmentation for Unbalanced Emotion Datasets

Learning with small or highly unbalanced datasets have long been an important area of research in computer vision. Extracting features from these datasets is more difficult and the training process of deep learning models could be hindered. In order to solve this problem many approaches have been proposed, among which data augmentation is widely used since it can balance the dataset by generating new samples for small classes. Current data augmentation methods like image manipulation can only generate samples with low diversity, generative model based methods can not guarantee the quality and distribution of generated images. This paper proposes an offline data augmentation method based on generative adversarial network (GAN) latent space manipulation to create new samples for small classes in unbalanced image datasets. This method helps fill the sparse data manifold of small classes with new samples and can guarantee data quantity, quality, diversity and distribution at the same time. More specifically, we choose the facial expression recognition (FER) task as an example where linear interpolation, combination and concatenation of StyleGAN latent codes are used to generate new samples. We further apply latent space vector arithmetic to perform inter-class translation for different facial expressions. The proposed methods are evaluated through extensive experiments on two different FER databases.