Siamese Contrastive Embedding Network for Compositional Zero-Shot Learning

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen compositions formed from seen state and object during training. Since the same state may be various in the visual appearance while entangled with different objects, CZSL is still a challenging task. Some methods recognize state and object with two trained classifiers, ignoring the impact of the interaction between object and state; the other methods try to learn the joint representation of the state-object compositions, leading to the domain gap between seen and unseen composition sets. In this paper, we propose a novel Siamese Contrastive Embedding Network (SCEN)¹¹Code: https://github.com/XDUxyLi/SCEN-master for unseen composition recognition. Considering the entanglement between state and object, we embed the visual feature into a Siamese Contrastive Space to capture prototypes of them separately, alleviating the interaction between state and object. In addition, we design a State Transition Module (STM) to increase the diversity of training compositions, improving the robustness of the recognition model. Extensive experiments indicate that our method significantly outperforms the state-of-the-art approaches on three challenging benchmark datasets, including the recent proposed C-QGA dataset.

1. Introduction

Humans possess the ability to compose their knowledge of known entities to generalize to novel concepts inherently. Given words, such as green horse, people can combine the known state green with the known object horse immediately, although they have never seen the inexistent stuff. To equip an AI system the similar ability, Compositional Zero-Shot Learning (CZSL) [20] is proposed, which aims to recognize unseen compositions composed of a set of seen states and objects. In CZSL setting, each composition comprises two components, namely, state and object, where the compositions of train and test sets are disjoint. Figure 1.

The overall concept of our method. We aim to separately extract discriminative prototypes of state and object based on establishing state and object specific databases, which can generalize to represent corresponding properties.