Knowledge Helps Pretrained Model: An Ensemble Model of Knowledge Model and CLIP for Zero-shot Image Recognition

The zero-shot image classification task is to recognize image without training samples. Knowledge based method is explored in this field, for knowledge graph has the advantage of utilize explicit knowledge to transfer knowledge from seen set to unseen set. Recently, large scale pretrained model is demonstrated to have more powerful image-text alignment capabilities which is help for zero-shot image classification task. However, the pretrained model is trained without explicit knowledge. Therefore, in this paper, we propose an ensemble method Knowledge and Pretrained Ensemble (KPE) model that utilize the strength of both of the knowledge graph and large scale pretrained model. Experimental results show the effectiveness of our model.

I. Introduction

In recent years, image recognition technology based on deep learning has gained extensive attention. This technology has applications in many industrial fields, such as medical imaging [1], security systems [2], and fault detection [3]. Image recognition technology based on deep learning has also gained extensive attention in academia. However, with the development of technology, researchers gradually found that deep learning technology requires a large amount of data support. However, labeling and collecting data requires a lot of manpower and money. Therefore, many researchers have begun to study deep learning models that do not require relevant data support and can also have high performance on image recognition tasks. This is the zero-shot image recognition [4].