Modernizing Old Photos Using Multiple References via Photorealistic Style Transfer

This paper firstly presents old photo modernization using multiple references by performing stylization and enhancement in a unified manner. In order to modernize old photos, we propose a novel multi-reference-based old photo modernization (MROPM) framework consisting of a network MROPM-Net and a novel synthetic data generation scheme. MROPM-Net stylizes old photos using multiple references via photorealistic style transfer (PST) and further enhances the results to produce modern-looking images. Meanwhile, the synthetic data generation scheme trains the network to effectively utilize multiple references to perform modernization. To evaluate the performance, we propose a new old photos benchmark dataset (CHD) consisting of diverse natural indoor and outdoor scenes. Extensive experiments show that the proposed method outperforms other baselines in performing modernization on real old photos, even though no old photos were used during training. Moreover, our method can appropriately select styles from multiple references for each semantic region in the old photo to further improve the modernization performance.

1. Introduction

Old photos taken a long time ago may contain important information that carry cultural and heritage values, e.g., photos of Queen Elizabeth II's coronation. Such old images may contain multiple degradations, e.g., scratches, and old photo artifacts, e.g., color fading, often preventing peo-ple from understanding the scene. To restore these images, a skilled expert needs to perform laborious manual pro-cesses such as degradation restoration and modernization, i.e., colorization or enhancement, to make them look mod-ern [44]. Consequently, early studies [8], [39] try to restore damaged old photos automatically by using traditional in-painting techniques. However, solely re-synthesizing dam-aged regions in the image is inadequate to ensure old photos look modern, as the overall sty Ie remains similar.