Scene Designer: a Unified Model for Scene Search and Synthesis from Sketch

Scene Designer is a novel method for searching and generating images using free-hand sketches of scene compositions; i.e. drawings that describe both the appearance and relative positions of objects. Our core contribution is a single unified model to learn both a cross-modal search embedding for matching sketched compositions to images, and an object embedding for layout synthesis. We show that a graph neural network (GNN) followed by Transformer under our novel contrastive learning setting is required to allow learning correlations between object type, appearance and arrangement, driving a mask generation module that synthesizes coherent scene layouts, whilst also delivering state of the art sketch based visual search of scenes.

1. Introduction

Creativity is increasingly inspired by the wealth of visual content online. Visual search eases content discovery and re-use, whilst generative artwork is emerging as a novel genre, driven by models trained on thousands of images. Yet the fusion of search and synthesis is under-explored. Generative content oveoffers users control but rarely the quality and diversity of real images. By contrast, search offers quality but not customization. Recent works have explored both image search and generation guided by freehand sketches; an intuitive way to communicate visual intent. Sketch therefore offers an opportunity to unify search and synthesis technologies within a creative workflow.