What Topics Do Images Say: A Neural Image Captioning Model with Topic Representation

Image captioning aims to generate descriptions of images with natural language sentences automatically. Most methods tackle this problem in an end-to-end fashion in recent years, which generates captions directly from image level features but ignores high-level semantic information. The method that introduced attribute concept into the CNN-RNN framework made a considerable improvement while the performance depended on the manually selected attributes heavily. In this paper, we propose a topic-guided neural image captioning model which incorporates a topic model into the CNN-RNN framework. Our model represents each image as a set of topics and each topic as various words with relevant distributions. We conduct experiments on Microsoft COCO dataset. The results show that our model outperforms the baselines and achieves promising performance. It verifies that the topic features are effective to represent high-level semantic information of images.