Visual Question Answering Using Deep Learning

Visual Question Answering (VQA) in recent times challenges fields that have received an outsized interest from the areas of Natural Language Processing and Computer Vision. VQA aims to establish an intelligent system to predict the answers for the natural language questions raised related to the image. The questions about the abstract or real word images are appealed to the VQA system; The system understands the image, and questions using Natural Language Processing (NLP) and Computer Vision which aims to predict the answer in natural language. The main issues which affect the performance of the VQA system is the inability to deal with the open-ended question acquired from the user. The proposed system is developed with a Graphical User Interface (GUI) that extracts the image features using pretrained VGG 16, and Golve embedding and Long Short- Term Memory (LSTM) are used in order to extract question features. By merging the characteristics of the images and the questions using pointwise multiplication the ultimate result is obtained. The acquired result is passed through a softmax layer to find the top 5 predictions about the image question. The proposed system has been experimented with various open-ended questions to show the robustness of the system. VQA finds its application in various real-world scenarios such as self-driving cars and guiding visually impaired people. Visual questions aim different parts of an image, including underlying context and background details.

I. Introduction

VQA[1] is a multi-discipline research problem that gained popularity from academic communities like natural language processing and computer vision because humans when looked at images, tend to see objects and understand how they interact with their properties and states. Visual question answering (VQA) is fascinating as it permits humans to recognize models they originally see[3]. The current approaches in VQA or grounding rely on concatenating vectors or applying elementwise sum or product. Researchers have utilized many techniques to build a VQA model. Deep learning methods can attain advanced results on challenging face recognition, object detection and object classification which are parts of computer vision. Deep learning algorithms have shown propitious results for VQA rather than other handcrafted algorithms and the sample example for the VQA system is shown in Fig. 1. For extracting the image features, the CNN algorithm in deep learning is used. Fig. 1.

Example for vqa system