Adaptive temporal fusion network with depth supervision and modulation for robust three-dimensional object detection in complex scenes

Autonomous driving perception relies on cameras and Light Detection and Ranging (LiDAR) sensors. Existing methods for LiDAR-camera fusion are primarily based on the Lift-Splat (LS) framework, which serves as a foundation for multi-modal fusion. However, these methods still face challenges such as unreliable depth information, insufficient dynamic perception, and limited robustness. This paper proposes a novel multi-modal three-dimensional (3D) detection method that optimizes depth by fully leveraging image and point cloud data and employs spatiotemporal deformable attention for adaptive fusion across frames. Specifically, we generate optimized depth maps through point clouds for depth supervision, refine the depth using Conditional Random Fields (CRF), and improve the fusion features by optimizing the depth estimation range. Additionally, we propose a dual-alignment method with spatiotemporal adaptive attention to acquire high-quality temporal features, allowing the model to learn beneficial information from adjacent frames. The proposed method achieves leading mean Average Precision (mAP) on mainstream 3D object detection datasets. Extensive experiments on multiple datasets demonstrate the superiority of the proposed method. Notably, our method remains effective even when a sensor fails, highlighting its potential to improve the robustness of autonomous perception in real-world scenarios.