Enhanced Multi-Modal Knowledge Graphs Representation Learning via Self-Attention Fusion

The complexity and diversity of multi-modal knowledge graphs arise from their ability to fuse information from multiple sources. Traditional knowledge graph representation learning faces challenges in effectively capturing and efficiently processing the complexity of multi-modal knowledge graphs. Moreover, conventional methods primarily rely on embeddings, overlooking the richness of multi-modal information, further diminishing the effectiveness of representation learning. To address these issues, this study proposes an Enhanced Multi-modal Knowledge Graph Representation Learning model (MATUCKER) based on the core of TuckER, which adaptively fuses multi-modal information using multi-head self-attention. In this approach, we employ the ResNet model to encode image features of multi-modal entities. These encoded features are then concatenated with the semantic embeddings of multi-modal entities and fed into a multi-head self-attention mechanism for fusion. The resulting multi-modal entity feature encodings are subsequently input into the TuckER model for representation learning. The model's performance is evaluated on the publicly available OpenBG-IMG dataset for link prediction tasks. The experimental results clearly demonstrate that our proposed model significantly outperforms both traditional methods and the multi-modal representation learning model, TransAE, in terms of multi-modal representation learning performance. We achieved significant improvements in key performance metrics compared to the TuckER backbone: the Mean Reciprocal Rank (MRR) increased by 3%, the HITS@10 metric improved by 5%, and the Mean Rank (MR) decreased by approximately 1,000.