An Unsupervised Vision-related Keywords Retrieval and Fusion Method for Visual Storytelling

Visual storytelling is a multi-modal generation task aiming to generate a coherent story for a sequence of images. Previous visual storytelling models utilize task-beneficial non-visual features, e.g., emotion, sentiment, or knowledge graph, as additional supplements for visual features to further improve the generation quality of stories. However, these appropriate non-visual features require to be carefully designed and selected by specialized researchers, moreover, high-quality external knowledge sources are not readily available. This increases the development cost of the VST model. To alleviate the above problem, this paper explores mining the knowledge existing in the multi-modal pre-trained models (MM-PTMs). First, we propose an Unsupervised Keywords Retrieval module (UKR), which takes an MM-PTM as the expert to select image-related keywords from a prepared task-related text corpus. The retrieved keywords not only complement and illustrate the visual features of the images but also provide more explicit generative signals to improve the interpretability and controllability of the generation process. Furthermore, we propose a Local Multi-modal Adaptive Fusion module (LMAF) to better fuse the textual and visual features and avoid noise brought by irrelevant keywords. LMAF dynamically aggregates features from both modalities through finer-grained correlation matching. The experimental results on the VST dataset, VIST, show that our proposed method achieves competitive results on several automatic metrics. Comparable results to previous methods can be achieved even if the model does not refer to visual features during story generation.