CNN-Based Metrics for Performance Evaluation of Generative Adversarial Networks

In this work, we propose two convolutional neural network (CNN) based metrics, classification score (CS) and distribution score (DS), for performance evaluation of generative adversarial networks (GANs). Though GAN-generated images can be evaluated through manual assessment of visual fidelity, it is prolonged, subjective, challenging, tiresome, and can be misleading. Existing quantitative methods are biased toward memory GAN and fail to detect overfitting. CS and DS allow us to experimentally prove that training of GANs is actually guided by the dataset that it improves with every epoch and gets closer to following the distribution of the dataset. Both methods are based on GAN-generated image classification by CNN. CS is the root mean square (rms) value of three different classification techniques, direct classification (DC), indirect classification (IC), and blind classification (BC). It exhibits the degree to which GAN can learn the features and generate fake images similar to real datasets. DS shows the contrast between the mean distribution of GAN-generated data and the real data. It indicates the extent to which GANs can create synthetic images with similar distribution to real datasets. We evaluated CS and DS metrics for different variants of GANs and compared their performances with existing metrics. Results show that CS and DS can evaluate the different variants of GANs quantitatively and qualitatively while detecting overfitting and mode collapse.

I. Introduction

Adversarial process enables us to learn implicit generative models known as generative adversarial networks (GANs) [1]. GANs can be trained through both semisupervised and unsupervised learning. The fundamental idea of a GAN model is based on a two-person min-max zero-sum game. There are two different networks, the generator, and the discriminator, in a GAN model that correspond to the game's two players. These players compete with one another. Real data are not accessible to the generator, and it tries to generate fake data similar to real data from noise. Both real and fake data are accessible to the discriminator, which distinguishes between the two and gives feedback to the generator to learn the features and distribution of real images [2]. Both the models are trained simultaneously until the discriminator cannot distinguish data produced by the generator and real data. These two networks are usually convolutional networks or fully connected layers [3].