Image Captioning with Emotional Information via Multiple Model

The area of image captioning has been significantly promotion on the quality of generated sentence in recent years. However, machine generated sentences often lack emotional or sentimental dimension and lose a lot of high-level sentiment information in CNN stage, which may be a clue understanding the state of the objects. To improve that, we propose a new captioning architecture (e-LSTM), which designed to integrates the experience learned from multiple models trained on different datasets. In order to explore the influence of emotional features on target recognition, we designed 4 variants of models. Experiments on COCO image captioning dataset prove that the sentences generated by our model are more vivid and the evaluation of VAEDR quantifies this advantage. Most remarkably, the promotion on classic evaluation matrix proves our method can use the additional input as supplementary information to capture the state of objects in the image more accurately.