Semantic Information Reasoning and Multi-Step Cross-Modal Interaction Network for Image-Text Retrieval

Image-text retrieval is a basic cross-modal task, and the basic idea is to learn image-text matching. The key to this task is to accurately measure the similarity between these two modalities. However, most existing methods tend to measure image-text similarity based on highly correlated region-word pairs, which lacks the ability to capture complex cross-modal semantic associations, thereby resulting in failure to distinguish hard negative examples that are similar in semantic content. Furthermore, existing methods focus more on alignment of different modal features, thus neglecting deep reasoning about the semantic concepts within the modality. In this paper, we propose a new semantic information reasoning and multi-step cross-modal interaction (SRMCI) network to simultaneously explore intra-modal and inter-modal relationships. Specifically, we first propose a semantic information reasoning module to fully explore potential semantic relationships within modalities and focus on fragments with important semantic information. Then, we design a multi-step cross-modal interaction module that utilizes two different inter-modal interactive attentions and an effective gated fusion strategy to capture fragment-level and fragment-instance-level cross-modal semantic associations. In particular, this gated fusion strategy is used to learn better fine-grained feature representations to aid in understanding and establishing semantic associations between different modalities. Experiments on the benchmark datasets MS-COCO and Flickr30K demonstrate the effectiveness of our method.