Efficient Unified Demosaicing for Bayer and Non-Bayer Patterned Image Sensors

As the physical size of recent CMOS image sensors (CIS) gets smaller, the latest mobile cameras adopt unique non-Bayer color filter array (CFA) patterns (e.g., Quad, Nona, Q×Q), which consist of homogeneous color units with adjacent pixels. These non-Bayer CFAs are superior to conventional Bayer CFA thanks to their changeable pixel-bin sizes for different light conditions, but may introduce visual artifacts during demosaicing due to their inherent pixel pattern structures and sensor hardware characteristics. Previous demosaicing methods have primarily focused on Bayer CFA, necessitating distinct reconstruction methods for non-Bayer CIS with various CFA modes under different lighting conditions. In this work, we propose an efficient unified demosaicing method that can be applied to both conventional Bayer RAW and various non-Bayer CFAs’ RAW data in different operation modes. Our Knowledge Learning-based demosaicing model for Adaptive Patterns, namely KLAP, utilizes CFA-adaptive filters for only 1% key filters in the network for each CFA, but still manages to effectively de-mosaic all the CFAs, yielding comparable performance to the large-scale models. Furthermore, by employing meta-learning during inference (KLAP-M), our model is able to eliminate unknown sensor-generic artifacts in real RAW data, effectively bridging the gap between synthetic images and real sensor RAW. Our KLAP and KLAP-M methods achieved state-of-the-art demosaicing performance in both synthetic and real RAW data of Bayer and non-Bayer CFAs.

1. Introduction

Demosaicing (DM) is the process of interpolating single-channel input images into RGB output images within an embedded Image Signal Processor (ISP). With the growing demand for high-quality mobile camera images, CMOS image sensor (CIS) resolution has increased dramatically, even reaching 200 million pixels in the latest smartphones. However, as image sensors cannot infinitely increase in size, pixel size has been reduced to enhance resolution. Smaller CISs are more vulnerable to noise and degradation in image restoration capabilities because they are more sensitive to variations in light reception, especially in low-light condition [13], [23], [38], [39]. As a result, modern high-end smartphones have started using image sensors that group adjacent homogeneous pixels, resulting in non-Bayer Quad, Nona, and Quad-by-Quad (Q×Q) sensors [19], [38], [41], while still retaining some of the properties of the standard Bayer CFA [5] pattern. Quad, Nona, and Q×Q sensors combine the same color pixel arrays of 2×2, 3×3, and 4×4 respectively, resulting in homogeneous pixel units (i.e., Gr, R, B, and Gb) for each sensor, as shown in Fig. 1(a).