Scene Graph based Fusion Network for Image-Text Retrieval

A critical challenge to image-text retrieval is how to learn accurate correspondences between images and texts. Most existing methods mainly focus on coarse-grained correspondences based on co-occurrences of semantic objects, while failing to distinguish the fine-grained local correspondences. In this paper, we propose a novel Scene Graph based Fusion Network (dubbed SGFN), which enhances the images’/texts’ features through intra-and cross-modal fusion for image-text retrieval. To be specific, we design an intra-modal hierarchical attention fusion to incorporate semantic contexts, such as objects, attributes, and relationships, into images’/texts’ feature vectors via scene graphs, and a cross-modal attention fusion to combine the contextual semantics and local fusion via contextual vectors. Extensive experiments on public datasets Flickr30K and MSCOCO show that our SGFN performs better than quite a few SOTA image-text retrieval methods.

I. Introduction

Image-text retrieval is one of the fundamental tasks in the field of vision and language [6]. Its goal is to effectively retrieve the most similar samples to its content from the database of image (text) modality given a query of text (image) modality. The biggest challenge is to narrow the semantic gap between cross-modal data for accurate similarity of image-text pairs.