Enhance Training Objectives for Image Captioning with Decomposed Sequence-level Metric

Image captioning aims to generate fluent and accurate descriptions for images. To evaluate the quality of captions, various metrics have been proposed. However, current metrics only assess captions at sequence-level, which overwhelms the distinctions between each token. Thus, existing objectives tend to treat each token equally, assigning them with identical weights in loss functions. Intuitively, key words in a caption carry the primary information and contribute more than other words to sequence-level metrics. They should be distinguished and weighted more during training. In this work, we propose to explicitly measure each word and guide the model to focus more on key words in captions. Firstly, we devise token-level CIDEr (CIDEr-T) as a new metric to quantify the importance of each word, by decomposing the sequence-level CIDEr into token-level granularity. CIDEr-T maintains consistency with CIDEr and shows the distinctions between tokens. Thus, we engage CIDEr-T scores of each token as their unique weights in the raw loss functions, which can bridge the gap between training and evaluation.