A Calibration and Real-Time Object Matching Method for Heterogeneous Multi-Camera System

The heterogeneous multi-camera system composed of thermal-infrared cameras and visible spectrum (VS) cameras can better perceive the surrounding information in complex environments, so it is widely used in many intelligent unmanned devices, such as unmanned aerial vehicles (UAV) and patrol robots. In this article, a stereo calibration strategy is proposed for this vision system, and a real-time target matching method for heterogeneous images is proposed based on double epipolar constraint. First, a new calibration board is designed, and a calibration points detection and sorting algorithm is proposed. Second, the thermal-infrared camera and VS camera are calibrated. Then, the relative position and posture between the two cameras and the projection relationship between the heterologous images are obtained by stereo calibration using the adjusted images and applied to stereo vision. Finally, the epipolar constraint is applied to the target matching under the heterogenous image combined with the existing target detection technology. In experiments, the average relative error of calibration point positioning using a heterogeneous vision system is 0.0209, which is only 0.0014 different from that using a homogeneous vision system. When the target appears in the image as a whole and is detected, target matching in a pair of heterologous images only takes 0.068 s and the accuracy is 96%, which meets the real-time and accuracy requirements of general application environments.