A Differentiable Two-stage Alignment Scheme for Burst Image Reconstruction with Large Shift

Denoising and demosaicking are two essential steps to reconstruct a clean full-color image from the raw data. Recently, joint denoising and demosaicking (JDD) for burst images, namely JDD-B, has attracted much attention by using multiple raw images captured in a short time to reconstruct a single high-quality image. One key challenge of JDD-B lies in the robust alignment of image frames. State-of-the-art alignment methods in feature domain cannot effectively utilize the temporal information of burst images, where large shifts commonly exist due to camera and object motion. In addition, the higher resolution (e.g., 4K) of modern imaging devices results in larger displacement between frames. To address these challenges, we design a differentiable two-stage alignment scheme sequentially in patch and pixel level for effective JDD-B. The input burst images are firstly aligned in the patch level by using a differentiable progressive block matching method, which can estimate the offset between distant frames with small computational cost. Then we perform implicit pixel-wise alignment in full-resolution feature domain to refine the alignment results. The two stages are jointly trained in an end-to-end manner. Extensive experiments demonstrate the significant improvement of our method over existing JDD-B methods. Codes are available at https://github.com/GuoShi28/2StageAlign.

1. Introduction

Color demosaicking and denoising are two essential steps in digital camera imaging pipeline to reconstruct a high quality full-color image from the sensor raw data. Color demosaicking [9], [20], [29], [45], [46] recovers the missing color components from the color-filter-array (CFA) data collected by the single-chip CCD/CMOS sensor, while denoising [21], [31], [48], [49] removes the noise in image data caused by the photon arrival statistics and the imprecision in readout circuitry. Since the two tasks are actually correlated and can be performed jointly, many joint denoising and demosaicking (JDD) algorithms have been developed [5], [11], [14], [15], [23], [28], which are however focused on single image restoration. With the prevalent use of smart-phone cameras [1], it becomes crucial to restore images from data with low signal-to-noise ratio due to the small sensor and lens of smartphone cameras. To this end, performing JDD with burst images (JDD-B) has become increasingly popular and important in recent years [13].