Deep SR-HDR: Joint Learning of Super-Resolution and High Dynamic Range Imaging for Dynamic Scenes

The visual quality of a single image captured by a digital camera usually suffers from limited spatial resolution and low dynamic range (LDR) due to sensor constraints. To address these problems, recent works have independently applied convolutional neural networks (CNNs) to super-resolution (SR) and high dynamic range (HDR) imaging and made significant improvements in visual quality. However, directly connecting SR and HDR networks is an inefficient way to enhance image quality, because these two tasks share most of the same processing steps. To this end, we propose a deep neural network for the joint task of SR and HDR imaging, termed Deep SR-HDR, which reconstructs a high-resolution (HR) HDR image from a set of differently exposed low-resolution (LR) LDR images of a dynamic scene. Specifically, we merge the shared processing steps, including feature extraction and alignment of these two tasks. In particular, to handle large-scale complex motions, we design a multi-scale deformable module (MSDM) that estimates the sampling location offsets in a coarse-to-fine manner and then flexibly integrates useful information to compensate for the missing content in the motion regions. Then, we divide the fusion stage into two branches for HDR generation and high-frequency information extraction. With the cooperation and interactions of these modules, the proposed network reconstructs high-quality HR HDR images. Extensive qualitative and quantitative experimental results demonstrate the superiority and high efficiency of the proposed network.

I. Introduction

Currently, digital images captured by cameras are significantly important in life, such as people taking photos to record their daily lives. However, due to the inherent limitations of camera sensors, captured images actually cannot portray the fine details of real scenes. In particular, the demands for spatial resolution and dynamic range are increasing. On the one hand, with the restriction of spatial sampling, the camera captures LR images that lose sub-pixel details and high-frequency information compared to high resolution (HR) images. This problem can be solved by reducing the pixel size or increasing the chip size. However, these solutions will yield more shot noise and require a higher cost [1]. On the other hand, due to the limited potential well capacity of the image sensors [2], the camera can obtain only LDR images that lack enough details in the under/over-exposed regions compared to high dynamic range (HDR) images. Although some hardware-based enhancements [3], [4] can produce HDR images, they are too expensive to be widely adopted. Therefore, owing to the demand for HR HDR images, it is a practical way to enhance the image quality in both aspects through software solutions.