Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications

Recent modern displays are now able to render high dynamic range (HDR), high resolution (HR) videos of up to 8K UHD (Ultra High Definition). Consequently, UHD HDR broadcasting and streaming have emerged as high quality premium services. However, due to the lack of original UHD HDR video content, appropriate conversion technologies are urgently needed to transform the legacy low resolution (LR) standard dynamic range (SDR) videos into UHD HDR versions. In this paper, we propose a joint super-resolution (SR) and inverse tone-mapping (ITM) framework, called Deep SR-ITM, which learns the direct mapping from LR SDR video to their HR HDR version. Joint SR and ITM is an intricate task, where high frequency details must be restored for SR, jointly with the local contrast, for ITM. Our network is able to restore fine details by decomposing the input image and focusing on the separate base (low frequency) and detail (high frequency) layers. Moreover, the proposed modulation blocks apply location-variant operations to enhance local contrast. The Deep SR-ITM shows good subjective quality with increased contrast and details, outperforming the previous joint SR-ITM method.

1. Introduction

Modern TVs come with 4K/8K UHD (Ultra High Definition) displays and high dynamic range (HDR) capabilities. Nevertheless, the current Digital TV and Internet TV (IPTV) services still provide the legacy video contents of Full HD (FHD) resolution and standard dynamic range (SDR), which then, must be rendered on the premium TV displays that support 4K/8K UHD and HDR videos. Therefore at the terminal end, it is necessary to convert the FHD SDR videos to 4K/8K UHD HDR in order to display them on the premium displays. Furthermore, the new media services of high quality suffer from the lack of original 4K/8K UHD and HDR visual content. Thus, it is also essential to convert the legacy contents of Full HD and SDR video to 4K/8K UHD and HDR videos at the content production end.