CLIPtone: Unsupervised Learning for Text-Based Image Tone Adjustment

Recent image tone adjustment (or enhancement) approaches have predominantly adopted supervised learning for learning human-centric perceptual assessment. However, these approaches are constrained by intrinsic challenges of supervised learning. Primarily, the requirement for expertly-curated or retouched images escalates the data acquisition expenses. Moreover, their coverage of target styles is confined to stylistic variants inferred from the training data. To surmount the above challenges, we propose an unsupervised learning-based approach for text-based image tone adjustment, CLIPtone, that extends an existing image enhancement method to accommodate natural language descriptions. Specifically, we design a hyper-network to adaptively modulate the pretrained parameters of a back-bone model based on a text description. To assess whether an adjusted image aligns with its text description without a ground-truth image, we utilize CLIP, which is trained on a vast set of language-image pairs and thus encompasses the knowledge of human perception. The major advantages of our approach are threefold: (i) minimal data collection expenses, (ii) support for a range of adjustments, and (iii) the ability to handle novel text descriptions unseen in training. The efficacy of the proposed method is demonstrated through comprehensive experiments including a user study.

1. Introduction

Image tone adjustment aims for the alteration of the tonal properties of an image, including brightness, contrast, and color balance. It is also termed image tone enhancement, as it has primarily been exploited to enhance the image aes-thetics. It is essential for various applications ranging from photography to medical imaging, and the importance of effective tone adjustment techniques has grown significantly. Figure 1.

We present CLIPtone, a text-based image tone adjustment framework trained in an unsupervised manner. With its superior understanding of natural languages, CLIPtone is capable of performing successful adjustments across a range of text descriptions, including those previously deemed challenging.