PRNet: Pyramid Restoration Network for RAW Image Super-Resolution

Typically, image super-resolution (SR) methods are applied to the standard RGB (sRGB) images produced by the image signal processing (ISP) pipeline of digital cameras. However, due to error accumulation, low bit depth and the nonlinearity with scene radiance in sRGB images, performing SR on them is sub-optimal. To address this issue, a RAW image SR method called pyramid restoration network (PRNet) is proposed in this paper. Firstly, PRNet takes the low-resolution (LR) RAW image as input, and generates a rough estimation of the SR result in the linear color space. Afterwards, a pyramid refinement (PR) sub-network refines image details in the intermediate SR result and corrects its colors in a divide-and-conquer manner. To learn the appropriate colors for displaying, external guidance is extracted from the LR reference image in the sRGB color space, and then fed to the PR sub-network. To effectively incorporate the external guidance, the cross-layer correction module (CLCM), which fully investigates the long-range interactions between two input features, is introduced in the PR sub-network. Moreover, as different frequency components decomposed from the same image are highly correlated, in the PR sub-network, the refined features from a lower layer are utilized to support the feature refinement in an upper layer. Extensive experiments presented in this paper demonstrate that the proposed method is capable of recovering fine details and small structures in images while producing vivid colors that align with the output of a specific camera ISP pipeline.

I. Introduction

Image super-resolution (SR) aims to restore high-resolution (HR) images from given low-resolution (LR) images. Over the past few years, numerous convolutional neural network (CNN)-based SR methods have been proposed [1], [2], [3] and achieved remarkable performance. However, usually, the SR methods are applied to the standard RGB (sRGB) images produced by the image signal processing (ISP) pipeline of digital cameras [4], thus leading to the following three main drawbacks: 1)

Error accumulation: Image restoration operations, including color demosaicking [5], [6], [7], [8], [9] and noise reduction [10], [11], [12] (and sometimes deblurring), have been applied in the ISP pipeline [13]. Moreover, sRGB images are usually compressed and stored, e.g., in JPEG format, and thus inevitably brings quantization noises and compression artifacts. Since these operations are processed separately, the errors produced by each operation may accumulate and spread within sRGB images, potentially affecting the SR method.

2)

Low bit depth: Typically, in sRGB images, only 8 bits are used to record one color channel of a single pixel. Compared to a higher bit depth, such as 12 or 14 bits, the relatively low bit depth used in sRGB images limits the recorded visual information and thus considerably restricts the quality of SR results.

3)

Nonlinearity with scene radiance: In a common ISP pipeline, nonlinear operations, such as tone curve, look-up table (LUT) and gamma transformation, are applied to convert the colors from the linear color space to the nonlinear sRGB space [14], [15]. As a result, the pixel intensity in sRGB images is no longer linear to the scene radiance [16], which makes the SR task even more challenging.