Sound-Guided Semantic Image Manipulation

The recent success of the generative model shows that leveraging the multi-modal embedding space can manipu-late an image using text information. However, manipulating an image with other sources rather than text, such as sound, is not easy due to the dynamic characteristics of the sources. Especially, sound can convey vivid emotions and dynamic expressions of the real world. Here, we propose a framework that directly encodes sound into the multi-modal (image-text) embedding space and manipulates an image from the space. Our audio encoder is trained to pro-duce a latent representation from an audio input, which is forced to be aligned with image and text representations in the multi-modal embedding space. We use a direct latent op-timization method based on aligned embeddings for sound-guided image manipulation. We also show that our method can mix different modalities, i.e., text and audio, which en-rich the variety of the image modification. The experiments on zero-shot audio classification and semantic-level image classification show that our proposed model outperforms other text and sound-guided state-of-the-art methods.

1. Introduction

Image manipulation has been widely studied in the field of computer vision due to its usefulness in photo-realistic manipulation applications, social media image sharing, and image-based advertisement. An image can be used to trans-fer its style into the target image [14], [13]. Also, modifying specific parts in the human face image, such as hairstyle or color, is useful in image manipulation applications [49], [34]. The purpose of semantic image manipulation is to generate a novel image that contains both source image identification and semantic information of user intention. In this paper, we tackle the semantic image manipulation task, which is the task of modifying an image with user-provided se-mantic cues. To apply the user intention into the image, a mixture of sketches and text is used to perform image ma-nipulation and synthesis [33], [49]. User intention can be ap-plied by drawing a paint [33] or writing text with semantic meanings [49], [13]. Figure 1:

Modified images with sound-guided semantic image manipulation. Our method manipulates source images (top row) given user-provided sound (middle row) into semantic images (last row).