HKFNet: Fine-Grained External Knowledge Fusion for Fact-Based Visual Question Answering

Fact-based Visual Question Answering (F-VQA) aims to answer questions with observed images and external facts. The existing deep learning-based F-VQA methods still struggle to achieve a fine-grained selection when fusing external knowledge. Furthermore, there is a lack of joint learning between global contextual features and regional features relevant to questions. These limitations weaken the role of external knowledge and hinder the model’s ability to comprehend questions and answers. To address the aforementioned issues, we propose a Hierarchical Knowledge Fusion Network for the F-VQA task. The knowledge fusion comprises the Relation Layer, the Target Layer, and the Reasoning Layer, which aggregate valuable information from questions, images, and facts, respectively. Additionally, we design a Dual Semantic Relations Module to learn object-level contextual relations and pixel-level cross-modal relations simultaneously. Experimental results show that our model achieves state-of-the-art performance on the F-VQA task.