Boosting Spike Camera Image Reconstruction from a Perspective of Dealing with Spike Fluctuations

As a bio-inspired vision sensor with ultra-high speed, spike cameras exhibit great potential in recording dynamic scenes with high-speed motion or drastic light changes. Different from traditional cameras, each pixel in spike cam-eras records the arrival of photons continuously by firing binary spikes at an ultra-fine temporal granularity. In this process, multiple factors impact the imaging, including the photons' Poisson arrival, thermal noises from circuits, and quantization effects in spike readout. These factors intro-duce fluctuations to spikes, making the recorded spike in-tervals unstable and unable to reflect accurate light intensi-ties. In this paper, we present an approach to deal with spike fluctuations and boost spike camera image reconstruction. We first analyze the quantization effects and reveal the unbi-ased estimation attribute of the reciprocal of differential of spike firing time (DSFT). Based on this, we propose a spike representation module to use DSFT with multiple orders for fluctuation suppression, where DSFT with higher or-ders indicates spike integration duration between multiple spikes. We also propose a module for inter-moment feature alignment at multiple granularities. The coarser alignment is based on patch-level cross-attention with a local search strategy, and the finer alignment is based on deformable convolution at the pixel level. Experimental results demon-strate the effectiveness of our method on both synthetic and real-captured data. The source code and dataset are avail-able at https://github.com/ruizhao26/BSF.

1. Introduction

Vision technology has undergone remarkable advancements recently. Machine vision in scenes with high-speed motion or drastic light changes is a key challenge in emerging applications such as autonomous driving [14], unmanned aerial vehicles [72], and assistant referees in sports [18]. Traditional digital cameras typically record scenes with a frame rate of 30 Hz r-:» 120 Hz, which is inadequate to fulfill the demands of these applications. Figure 1.

Illustration of spike camera image reconstruction (SCIR) and comparison among recent methods [58], [64], [77] and ours. On the top-left, an orange point means a spike. Our method can better recover the textures. Please zoom in for more details.