Multi-Graph Based Hierarchical Semantic Fusion for Cross-Modal Representation

The main challenge of cross-modal retrieval is how to efficiently realize semantic alignment and reduce the heterogeneity gap. However, existing approaches ignore the multi-grained semantic knowledge learning from different modalities. To this end, this paper proposes a novel end-to-end cross-modal representation method, termed as Multi-Graph based Hierarchical Semantic Fusion (MG-HSF). This method is an integration of multi-graph hierarchical semantic fusion with cross-modal adversarial learning, which captures fine-grained and coarse-grained semantic knowledge from cross-modal samples, and generate modalities-invariant representations in a common subspace. To evaluate the performance, extensive experiments are conducted on three benchmarks. The experimental results show that our method is superior than the state-of-the-arts.

1. Introduction

With the advent of big data era, the amount of multi-modal data (e.g., image, text, audio, video, 3D model) in the Internet is growing explosively. This trend brings unprecedented challenges of accurate and efficient cross-modal retrieval [1]. As a hot-spot in the area of multimedia, cross-modal retrieval aims to find out objects of different modalities according to a query of a specific modality. This technology can be applied in many scenarios, such as multimedia search, recommendation system, VQA, etc.