Finding Patterns in Adversarial Training

Adversarial training has become an universally accepted robust method to train networks defending against adversarial attacks. However, feeding networks with adversarial examples is a cumbersome process, and there is no clear explanation for the differences between networks with the same architecture that are learned by adversarial training and learned by natural training method. So in this paper, we focus on the patterns of network learned by adversarial training. For comparison, we visualize the weights and feature maps of networks learned by these two training methods on MNIST, and find some patterns of adversarially trained model that are important for defending against adversarial examples and that are unique to naturally trained model. First, we find that adversarially trained network denoises the noisy data caused by adversarial images and enhances the outlines of semantically informative content in feature maps. Second, we find that adversarial training performs model compression, making some of the channels of a convolutional layer have no activation on input images. Further observation shows that different from naturally trained model, which have different most activation channel corresponding to different input, adversarially trained model often has the same most activation channel regardless of input. This phenomenon also leads to a sparse network reducing 50% number of parameters and 87% computation costs, and the pruned network performs as good as original network after pruning the least activation channels without retraining. But the naturally trained counterpart loses more than 40% accuracy using the same pruning strategy.