Transferable Semantic Augmentation for Domain Adaptation

Domain adaptation has been widely explored by transferring the knowledge from a label-rich source domain to a related but unlabeled target domain. Most existing domain adaptation algorithms attend to adapting feature representations across two domains with the guidance of a shared source-supervised classifier. However, such classifier limits the generalization ability towards unlabeled target recognition. To remedy this, we propose a Transferable Semantic Augmentation (TSA) approach to enhance the classifier adaptation ability through implicitly generating source features towards target semantics. Specifically, TSA is inspired by the fact that deep feature transformation towards a certain direction can be represented as meaningful semantic altering in the original input space. Thus, source features can be augmented to effectively equip with target semantics to train a more transferable classifier. To achieve this, for each class, we first use the inter-domain feature mean difference and target intra-class feature covariance to construct a multivariate normal distribution. Then we augment source features with random directions sampled from the distribution class-wisely. Interestingly, such source augmentation is implicitly implemented through an expected transferable cross-entropy loss over the augmented source distribution, where an upper bound of the expected loss is derived and minimized, introducing negligible computational overhead. As a light-weight and general technique, TSA can be easily plugged into various domain adaptation methods, bringing remarkable improvements. Comprehensive experiments on cross-domain benchmarks validate the efficacy of TSA.

1. Introduction

Deep learning has achieved remarkable success on various vision tasks, including image recognition [16], [12], [14] and semantic segmentation [55], [49]. However, the recent success of deep learning methods heavily relies on massive labeled data. In practice, collecting abundant annotated data is expensive [30], [6], [54], [33]. Meanwhile, each domain has its own specific exploratory factors, namely semantics, e.g., the illuminations, colors, visual angles or backgrounds, resulting in the domain shift [28]. Hence, traditional deep models trained on a large dataset usually show poor generalizations on a new domain due to the domain shift issues [24], [10]. To remedy this, one appealing alternative is domain adaptation (DA), which strives to leverage the knowledge of a label-rich source domain to assist the learning in a related but unlabeled target domain [24], [10].