PNRNet: Physically-Inspired Neural Rendering for Any-to-Any Relighting

Existing any-to-any relighting methods suffer from the task-aliasing effects and the loss of local details in the image generation process, such as shading and attached-shadow. In this paper, we present PNRNet, a novel neural architecture that decomposes the any-to-any relighting task into three simpler sub-tasks, i.e. lighting estimation, color temperature transfer, and lighting direction transfer, to avoid the task-aliasing effects. These sub-tasks are easy to learn and can be trained with direct supervisions independently. To better preserve local shading and attached-shadow details, we propose a parallel multi-scale network that incorporates multiple physical attributes to model local illuminations for lighting direction transfer. We also introduce a simple yet effective color temperature transfer network to learn a pixel-level non-linear function which allows color temperature adjustment beyond the predefined color temperatures and generalizes well to real images. Extensive experiments demonstrate that our proposed approach achieves better results quantitatively and qualitatively than prior works.

I. Introduction

The goal of this paper is to generate a relit image from the original RGB-D image to match the illumination setting of the given guide RGB-D image. As shown in Fig. 1, the inputs consist of a source image of a complex scene (Fig. 1a) and a guide image under novel lighting (Fig. 1b), and the output is the relit source image under the novel lighting provided by the guide image. Different from traditional image-based relighting where the target illumination is given explicitly [1]–[3], the illumination in any-to-any relighting is implicitly contained in the guide image. This has attracted much attention because it can benefit many applications when casual photographers do not have the expertise in lighting, but the images containing desired illumination settings can be easily obtained. Fig. 1.

Given one RGB-D image of a source scene (a) and one RGB-D image of a guide scene (b), our proposed method is able to relight the source scene with the illumination of the guide scene (d), while maintaining the local shading and attached-shadow details and accurately recovering the color temperature of the relit image (h).