ICTCAM: Introducing Convolution to Transformer-Based Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation(WSSS) is a challenging task, which only requires category information for segmentation prediction. Existing WSSS methods can be divided into two types: CNN-based and transformer-based, and the ways of generating pseudo labels are different. The former uses Class Activation Mapping(Cam)to generate pseudo labels, but there is a problem that the activated areas are concentrated in the most discriminative parts. The latter one choose to use attention map from the multi-head self-attention(MHSA) block, but there also exist the problems of significant background noise and incoherent object area. In order to solve the problems above, we propose ICTCAM to help transformer block obtain the ability of CNN, which include two modules named deeper stem(DStem) and convolutional feed-forward network(CFFN). The experiment results show that our modules have improved the performance of the network and achieve 69.9% mIoU, which is a new state-of-the-art performance on the PASCAL VOC 2012 dataset compared with similar networks.

I. Introduction

Semantic segmentation is a basic task in computer vision field, and its goal is to classify every pixel in the image. With the development of deep learning in recent years, the semantic segmentation algorithms have a great breakthrough, such as [1]–[4], and they have excellent performance on extensive datasets. However, most semantic segmentation algorithms train models in the fully supervised way, and it is a hard labor to assign each pixel the correct category. Because of this reason, many researchers now turn their attention to weakly supervised semantic segmentation(WSSS).