Zero-shot Diffusion Models for Demosaicing in Bayer and Non-Bayer Image Sensors

As the demand for high-quality, high-resolution images on mobile phones increases, modern devices now incorporate non-Bayer image sensors that maintain some characteristics of the traditional Bayer pattern while introducing new variations. This evolution has increased the complexity of restoring sensor raw images to full RGB using image signal processors (ISPs), revealing the limitations of conventional model-based ISP approaches. To address these challenges, several deep learning models have been proposed for raw image restoration. However, no existing research has utilized a pre-trained diffusion model in a zero-shot manner for this task, particularly across both traditional Bayer and newer non-Bayer sensors. We introduce a novel zero-shot diffusion model framework incorporating our degradation and pseudo-inverse degradation functions for the demosaicing task. Our approach is effectively and universally applicable to a wide range of sensor patterns, showing superior performance, particularly with the latest and future sensors where traditional methods struggle.

I. Introduction

CMOS image sensors, especially in mobile phone cameras, utilize a color filter array (CFA) pattern known as the Bayer pattern [1] for capturing raw input. This mosaiced single-channel sensor data is prone to inherent noise and significant color loss. The image signal processor (ISP) in mobile cameras processes and restores raw data, including converting the 1-channel CFA into the final RGB image through an interpolation process known as “demosaicing” [2].