1 Introduction
Recently, with the enormous advancement of deep learning, many domains like speech recognition [1] and visual object recognition [2], have achieved a dramatic improvement out of the state-of-the-art methods. Despite the great success, deep learning models have been proved vulnerable against perturbations. Specifically, Szegedy et al. [3] and Goodfellow et al. [4] have found that deep learning models may be easily fooled when a small perturbation (usually unnoticeable for humans) is applied to the images. The perturbed examples are also termed as “adversarial examples” [4].