Zero-Shot Learning With Attentive Region Embedding and Enhanced Semantics

The performance of zero-shot learning (ZSL) can be improved progressively by learning better features and generating pseudosamples for unseen classes. Existing ZSL works typically learn feature extractors and generators independently, which may shift the unseen samples away from their real distribution and suffers from the domain bias problem. In this article, to tackle this challenge, we propose a variational autoencoder (VAE)-based framework, that is, joint Attentive Region Embedding with Enhanced Semantics (AREES), which is tailored to advance the zero-shot recognition. Specifically, AREES is end-to-end trainable and consists of three network branches: 1) attentive region embedding is used to learn the semantic-guided visual features by the attention mechanism (AM); 2) a decomposition structure and a semantic pivot regularization are used to extract enhanced semantics; and 3) a multimodal VAE (mVAE) with the cross-reconstruction loss and the distribution alignment loss is used to obtain a shared latent embedding space of visual features and semantics. Finally, features’ extraction and features’ generation are optimized together in AREES to address the domain shift problem to a large extent. The comprehensive evaluations on six benchmarks, including the ImageNet, demonstrate the superiority of the proposed model over its state-of-the-art counterparts.

I. Introduction

Recently, the boundaries of standard classification tasks have been pushed further, partly due to the availability of deep learning techniques. Unfortunately, most classification models require a large number of labeled training examples, which is of great difficulty to annotate all the concepts with high quality because most categories of objects follow a long-tail distribution. To address this awkward situation, zero-shot learning (ZSL) [22] is proposed to recognize test samples that are not available in the training process. In the context of ZSL, seen and unseen classes are associated with semantic features that can be attributes [23] or word vectors [46]. ZSL completes the transferable recognition ability by building a relationship from seen classes to unseen classes with the help of shared semantic information. In real-world applications, the objects that need to be recognized may come from both seen and unseen classes instead of only from the unseen classes, which is called the generalized ZSL (GZSL) task.