Generating Questions via Unexploited OCR Texts: Prompt-Based Data Augmentation for TextVQA

Text-based Visual Question Answering (TextVQA) tasks rely on Optical Character Recognition (OCR) text to answer. There have been many models successfully exploring multi-modal features fusing and knowledge reasoning. However, current TextVQA datasets are few and the cost of using manual annotation is too high. So generating pseudo-labeled data is a better choice. In this paper, a prompt-based data augmentation method is proposed. The problems of current data augmentation are solved: 1) the distribution of the number of answer words in the pseudo-labeled data is not consistent with the real dataset. 2) the question forms in the pseudo-labeled data are not diverse. Specifically, prompt words are first matched to the constraints in the questions by finding the same words in the vocabulary. So, our generating model can generate different types of questions when the different prompt words are input. Experiments show that our method is significantly better than other state-of-the-art methods on TextVQA.

I. Introduction

The Visual Question Answering (VQA) task is an image-text multi-modal question answering task proposed by Agrawal in 2015 [1]. The task requires multimodal models with the ability to read, understand, fuse, and reason. In natural scenes, textual information appears in many places, such as car licenses, store names, and clothing logos. These OCR texts are indispensable supplementary information for visual question answering. This kind of task using OCR text is called Text-based Visual Question Answering [2]. An example is shown in Fig. 1.