Decomposed Soft Prompt Guided Fusion Enhancing for Compositional Zero-Shot Learning

Compositional Zero-Shot Learning (CZSL) aims to recognize novel concepts formed by known states and objects during training. Existing methods either learn the combined state-object representation, challenging the generalization of unseen compositions, or design two classifiers to identify state and object separately from image features, ignoring the intrinsic relationship between them. To jointly eliminate the above issues and construct a more robust CZSL system, we propose a novel framework termed Decomposed Fusion with Soft Prompt (DFSP)¹¹Code is available at: https://github.corn/Forest-art/DFSP.git, by involving vision-language models (VLMs)for unseen composition recognition. Specifically, DFSP constructs a vector combination of learnable soft prompts with state and object to establish the joint representation of them. In addition, a cross-modal decomposed fusion module is designed between the language and image branches, which decomposes state and object among language features instead of image features. Notably, being fused with the decomposed features, the image features can be more expressive for learning the relationship with states and objects, respectively, to improve the response of unseen compositions in the pair space, hence narrowing the domain gap between seen and unseen sets. Experimental results on three challenging benchmarks demonstrate that our approach significantly outperforms other state-of-the-art methods by large margins.

1. Introduction

Given an unseen concept, such as green tiger, even though this is a nonexistent stuff humans have never seen, they may associate the known state green with an image of tiger immediately. Inspired by this, Compositional Zero-Shot Learning (CZSL) is proposed with the purpose of equipping models with the ability to recognize novel concepts generated as humans do. Specifically, CZSL learns on visible primitive composed concepts (state and object) in the training phase, and recognizes unseen compositions in the inference phase. Figure 1.

The overview of dfsp. Our method aims to narrow the domain gap between seen and unseen compositions by fusing decomposed features and with image feature , while learn the joint representation between state and object in language branch. Being fused with the state and object features, image feature can learn the response of them respectively and improve the sensitiveness of unseen compositions.