Variational Contrastive Log Ratio Upper Bound of Mutual Information for Training Generative Models

Theoretically, a Generative adversarial network minimizes the Jensen-Shannon divergence between real data distribution and generated data distribution. This divergence is another form of mutual information between a mixture distribution and a binary distribution. It implies that we can build a similar generative model by optimizing the mutual information. This research proposes variational contrastive log-ratio upper bound vCLUB mutual information estimation on mixture distribution and the optimization algorithm to train two generative models. We call the models CLUB-sampling generative network (vCLUB-sampling GN) and vCLUB-non sampling generative network (vCLUB-non sampling GN). The results show that vCLUB-sampling outperforms GAN and vCLUB-non sampling GN on the MNIST dataset and has competitive results with GAN on the CIFAR-10 dataset. However, GAN outperforms vCLUB-non sampling GN on both datasets.