Event-Driven Heterogeneous Network for Video Deraining

Restoring clear frames from rainy videos poses a significant challenge due to the swift motion of rain streaks. Traditional frame-based visual sensors, which record dense scene content synchronously, struggle to capture the fast-moving rain information. Conversely, the novel bio-inspired event camera, known for its high temporal resolution and low latency, effectively records the motion trajectories of rapidly falling rain through asynchronously generated sparse event sequences. In light of these attributes, we introduce a novel event-driven convolutional spiking network for video deraining. For video restoration, we employ a Convolutional Neural Network as the backbone, extracting dense features from video sequences to map rain-soaked frames to clear ones. To remove rain, we meticulously design a bio-inspired Spiking Neural Network that adapts to the sparse event sequences, capturing features of falling rain. We then establish a bimodal feature fusion module that combines dense convolutional features with sparse spiking features. This fusion aids the backbone in accurately pinpointing rain streaks across spatiotemporal dimensions. Thus, our diverse network extracts and collaborates information from both events and videos, enhancing deraining performance. Experiments conducted on synthetic and real-world datasets prove that our network markedly surpasses existing video deraining techniques.