Interpreting Decision of Self-Attention Network’s in the Context of Image Recognition by Efficiently Utilizing Attention Scores

Deep Learning has risen to success exceeding human level in different tasks of computer vision in the last decade. Despite the gain in performance, the black box nature and lack of interpretation have always been a criticism against deep networks and in many cases a barrier to its application. Even though convolutional neural networks are the dominant, most successful architecture in vision problems, recently it has been shown that fully self-attention based networks can outperform convolutional networks in terms of performance and complexity. Despite being proposed initially in the language domain, self-attention based networks have been slowly finding their application and importance in vision problems as well. One of the key aspects of attention-based networks is the attention scores they provide. Visualizing attention scores to understand the inside of the deep network’s working is a common practice in different problems such as image captioning, machine translation, etc. In this paper, we propose an efficient framework to utilize attention scores to generate an explanation of the model’s decision when self-attention networks are implemented in the context of image recognition. Using our approach we show different visualizations of the original image and regions of the image that the network finds as important. We also compare the importance heatmap produced by our approach and other standard gradient-based approaches. We show that when the network assigns importance to a large cluster of region gradient-based methods fail to integrate the total pattern, a problem that our suggested strategy is able to solve.