Intelligent Painter: New Masking Strategy and Self- Referencing with Resampling

Painting with our own hands is not everyone’s talent. Some of us may dream big to create our own artwork but do not have the ability to do so. With the help of deep learning techniques, we nowadays can generate text-based painting. However, just typing text to create our own artwork is still different from doing it yourself (DIY). We proposed an application called intelligent painter, which can let users decide the placement of the objects and use the diffusion models to fill all the gaps after the users finish their placement. In this paper, we propose two major contributions to make better generation of images by (i) a new masking strategy and (ii) speeding up the process by 50% compared with resampling Denoising Diffusion Probabilistic Models (DDPM), with a self-pre-processing input step.

I. INTRODUCTION

Image generation makes use of deep learning approaches which have performed decent results in the past decade, including Generative Adversarial Networks (GANs) [1]–[3], Variational AutoEncoders (VAEs) [4]–[6] and Diffusion Models (DMs) [7], [8]. Without any conditioning, these generative models produce images from a latent space. The results depend highly on the training data and usually are limited to a specify type of scenes, such as human face or bedroom etc. Conditioning methods [8]–[10], no matter by texts or images, can guide the generation process to fit for the target result of the users. However, sometimes users may just want to choose specific objects and post them on preferred locations, just like drawing by their own hands.