Complementarity-Aware Space Learning for Video-Text Retrieval

In general, videos are powerful at recording physical patterns (e.g., spatial layout) while texts are great at describing abstract symbols (e.g., emotion). When video and text are used in multi-modal tasks, they are claimed to be complementary and their distinct information is crucial. However, when it comes to cross-modal tasks (e.g., retrieval), existing works usually use their common part in the form of common space learning while their distinct information is abandoned. In this paper, we argue that distinct information is also beneficial for cross-modal retrieval. To address this problem, we propose a divide-and-conquer learning approach, namely Complementarity-aware Space Learning (CSL), by recasting this challenge into learning of two spaces (i.e., latent and symbolic spaces) to simultaneously explore their common and distinct information by considering multi-modal complementary character. Specifically, we first propose to learn a symbolic space from video with a memory-based video encoder and a symbolic generator. In contrast, we also introduce learning a latent space from text with a text encoder and a memory-based latent feature selector. Finally, we propose a complementarity-aware loss by integrating two spaces to facilitate video-text retrieval tasks. Extensive experiments show that our approach outperforms existing state-of-the-art methods by 5.1%, 2.1% and 0.9% of R@10 for text-to-video retrieval on three benchmarks, respectively. Ablation study also verifies that the distinct information from video and text improves the retrieval performance. Trained models and source code have been released at https://github.com/NovaMind-Z/CSL.

I. Introduction

To date, multi-modal tasks involving videos and texts have attracted tremendous interest due to the explosive growth of short videos and the rapid development of deep learning, such as video question answering [1], [2], video grounding [3], and video captioning [4], [5]. These tasks aim at bridging a gap between human and machine intelligence by utilizing the complementary nature of videos and texts. When filling this gap, a core problem is: how to establish the potential correspondence between video and text, which is commonly referred to as video-text retrieval. So far, video-text retrieval is a challenging task due to the massive heterogeneity of video and text, resulting in considerable differences in their representation spaces. Therefore, great progress has been made in the past decades [6], [7], [8], [9], [10].