Learning Attribute-driven Disentangled Representations for Interactive Fashion Retrieval

Interactive retrieval for online fashion shopping provides the ability to change image retrieval results according to the user feedback. One common problem in interactive retrieval is that a specific user interaction (e.g., changing the color of a T-shirt) causes other aspects to change inadvertently (e.g., the retrieved item has a sleeve type different than the query). This is a consequence of existing methods learning visual representations that are semantically entangled in the embedding space, which limits the controllability of the retrieved results. We propose to leverage on the semantics of visual attributes to train convolutional networks that learn attribute-specific subspaces for each attribute to obtain disentangled representations. Thus operations, such as swapping out a particular attribute value for another, impact the attribute at hand and leave others untouched. We show that our model can be tailored to deal with different retrieval tasks while maintaining its disentanglement property. We obtain state-of-the-art performance on three interactive fashion retrieval tasks: attribute manipulation retrieval, conditional similarity retrieval, and outfit complementary item retrieval. Code and models are publicly available¹.

1. Introduction

Content-based image retrieval has been a fundamental computer vision task for decades. More recently, this task has evolved in the direction of enabling users to provide additional forms of interaction (e.g., sentences, attributes and clicks) along with the query image. Interactive image retrieval [11], [15], [57] is relevant in the context of online shopping, specifically for product categories for which appearance is one of the pre-eminent factors for selection, such as fashion items. In this context, it is not only necessary to train expressive visual representations of images [22], [33], [16], [42], [46], but also to empower the model with the ability of understanding interactions of the user and modify the search results accordingly.