Deep Relative Attributes

Relative attribute (RA) learning aims to learn the ranking function describing the relative strength of the attribute. Most of current learning approaches learn a linear ranking function for each attribute by use of the hand-crafted visual features. Different from the existing study, in this paper, we propose a novel deep relative attributes (DRA) algorithm to learn visual features and the effective nonlinear ranking function to describe the RA of image pairs in a unified framework. Here, visual features and the ranking function are learned jointly, and they can benefit each other. The proposed DRA model is comprised of five convolutional neural layers, five fully connected layers, and a relative loss function which contains the contrastive constraint and the similar constraint corresponding to the ordered image pairs and the unordered image pairs, respectively. To train the DRA model effectively, we make use of the transferred knowledge from the large scale visual recognition on ImageNet [1] to the RA learning task. We evaluate the proposed DRA model on three widely used datasets. Extensive experimental results demonstrate that the proposed DRA model consistently and significantly outperforms the state-of-the-art RA learning methods. On the public OSR, PubFig, and Shoes datasets, compared with the previous RA learning results [2], the average ranking accuracies have been significantly improved by about 8%, 9%, and 14%, respectively.

I. Introduction

Visual attributes are intrinsic properties in images with human-designed names (e.g., “natural,” “smiling”), and they are valuable as higher semantic cues than low level visual features in many interesting scenarios. For example, researchers have shown that visual attributes are valuable for facial verification [3], object recognition  [4]–[6], image retrieval/search [7]– [14], video retrieval and recommendation  [15], [16], generating descriptions of unfamiliar objects [17] and transfer learning  [18]–[21]. Many attributes mining and learning methods have also been proposed [22] –[24]. In these methods, the attributes are binary, which indicates the presence (or absence) of a certain property in an image. Compared with the binary attributes, using relative attributes (RA) is a much richer way for humans to describe objects semantically with relative visual properties. The consecutive relative values of the attributes can reflect not only whether the attribute appears in an image, but also the strength of the attribute. As a richer language of visual description than the commonly used binary attributes, RA learning has gained much attention and can be used in many applications especially social event analysis  [25]–[28], and zero-shot learning [2], [29] –[31].