Sketch-Based Shape Retrieval via Multi-view Attention and Generalized Similarity

Sketch-based shape retrieval has received increasing attention in computer vision and computer graphics. It suffers from the challenge gap between 2D sketches and 3D shapes. In this paper, we propose a generalized similarity matching framework based on a multi-view attention network (MVAN), which can retrieve 3D shape that is most similar to the query sketch. In proposed approach, firstly we compute 2D projections of 3D shapes from multiple viewpoints and utilize a convolutional neural network to extract low level feature maps of these 2D projections. Secondly a multi-view attention network is designed to fuse the feature maps and forms a more accurate 3D shape representation. Meanwhile we use a CNN to extract the feature of sketches. Thirdly the similarity between sketches and 3D shapes is estimated via a generalized similarity model, which fuses some traditional similarity model into a generalized form and optimizes its parameters using a data-driven method. Finally we combine the MVAN and generalized similarity model into a unified network and train the model in an end-to-end manner. The experimental results on SHREC'13 and SHREC'14 sketch track benchmark datasets demonstrate that the proposed method can outperform state-of-the-art methods.

I. Introduction

With the development of virtual reality, a large amount of 3D shapes have been created, which leads to a growing demand for effective retrieval of 3D shapes. There mainly three types of 3D shape retrieval methods: retrieval by keyword method, retrieval by 3D shape method and retrieval by sketch method. For keyword-based shape retrieval, users are supposed to provide several keywords to describe the feature of 3D shapes. Meanwhile the labels of 3D shapes should be rich and accurate enough. It not only increases the burden for labeling but is inefficient for users to query a 3D shape. The 3D-shape-based method seems more direct as a user can input a similar 3D shape and retrieve what he needs. However, obtaining a 3D shape required is not easy, so this approach is not practical in many situation. Compared to previous methods, sketch-based shape retrieval is more convenient, which make the sketch-based shape retrieval has been received considerable attention in recent years. For sketch-based shape retrieval, users could draw several lines to express their intents and do not need to be skillful in painting and 3D modeling. Fig. 1.

The sketches and 3D shapes drawn from SHREC’13 [1] dataset: (a) Sketches of book and fish, (b) 3D shapes of book and fish.