DemosaicFormer: Coarse-to-Fine Demosaicing Network for HybridEVS Camera

Hybrid Event-Based Vision Sensor (HybridEVS) is a novel sensor integrating traditional frame-based and event-based sensors, offering substantial benefits for applications requiring low-light, high dynamic range, and low-latency environments, such as smartphones and wearable devices. Despite its potential, the lack of Image signal processing (ISP) pipeline specifically designed for HybridEVS poses a significant challenge. To address this challenge, in this study, we propose a coarse-to-fine framework named DemosaicFormer which comprises coarse demosaicing and pixel correction. Coarse demosaicing network is designed to produce a preliminary high-quality estimate of the RGB image from the HybridEVS raw data while the pixel correction network enhances the performance of image restoration and mitigates the impact of defective pixels. Our key innovation is the design of a Multi-Scale Gating Module (MSGM) applying the integration of cross-scale features, which allows feature information to flow between different scales. Additionally, the adoption of progressive training and data augmentation strategies further improves model’s robustness and effectiveness. Experimental results show superior performance against the existing methods both qualitatively and visually, and our DemosaicFormer achieves the best performance in terms of all the evaluation metrics in the MIPI 2024 challenge on Demosaic for Hybridevs Camera. The code is available at this repository.

1. Introduction

Event-Based Vision Sensor (EVS) detects luminance changes asynchronously and will output event data immediately, which has the advantages of low power consumption and high sensitivity, and is suitable for capturing high dynamic range visual information without blurring. However, the inability to capture color information greatly limits the application scope of event cameras. Hybrid Event-based Vision Sensor (HybridEVS) [11] is a novel hybrid sensor formed by combining traditional frame-based sensor and event-based sensor. It combines the advantages of these sensors, offering high temporal resolution, low latency, and exceptional dynamic range while still capturing color information with higher Signal-to-Noise Ratio (SNR). Compared to traditional sensors, HybridEVS can perform better in a greater range of applications because of its hybrid design. Quad Bayer pattern, as shown in Fig. 1(a) is a common type of pattern widely employed in smartphone cameras due to its ability to obtain high-quality images under low light secnary by averaging four pixels within a 2 × 2 neighborhood. While signal-to-noise ratio (SNR) is improved in the binning mode, the spatial resolution is reduced as a tradeoff. Defect pixels are flaws caused by the sensor’s manufacturing process, where certain pixel values are inaccurate during the photoelectric conversion process.