BAPA-Net: Boundary Adaptation and Prototype Alignment for Cross-domain Semantic Segmentation

Existing cross-domain semantic segmentation methods usually focus on the overall segmentation results of whole objects but neglect the importance of object boundaries. In this work, we find that the segmentation performance can be considerably boosted if we treat object boundaries properly. For that, we propose a novel method called BAPA-Net, which is based on a convolutional neural network via Boundary Adaptation and Prototype Alignment, under the unsupervised domain adaptation setting. Specifically, we first construct additional images by pasting objects from source images to target images, and we develop a so-called boundary adaptation module to weigh each pixel based on its distance to the nearest boundary pixel of those pasted source objects. Moreover, we propose another prototype alignment module to reduce the domain mismatch by minimizing distances between the class prototypes of the source and target domains, where boundaries are removed to avoid domain confusion during prototype calculation. By integrating the boundary adaptation and prototype alignment, we are able to train a discriminative and domain-invariant model for cross-domain semantic segmentation. We conduct extensive experiments on the benchmark datasets of urban scenes (i.e., GTA5→Cityscapes and SYNTHIA→Cityscapes). And the promising results clearly show the effectiveness of our BAPA-Net method over existing state-of-the-art for cross-domain semantic segmentation. Our implementation is available at https://github.com/manmanjun/BAPA-Net.

1. Introduction

Because of the powerful representation ability of deep convolutional neural networks [26], it has deeply boosted the performance of the computer vision tasks including image recognition [45], [21], object detection [16], [32], semantic segmentation [33], [63], [3], etc. They all require plentiful images and accurate annotations to train high-performance models. Compared with image recognition, semantic segmentation is more complex and aims at classifying each pixel in an image. Therefore, collecting the annotations for segmentation is an extremely expensive and laborious process (e.g., 90 minutes per image for Cityscapes [7]).