Drone-based RGBT tiny person detection

RGBT person detection benefits numerous vital applications like surveillance, search, and rescue. Meanwhile, drones can capture images holding broad perspectives and large searching regions per frame, which can notably improve the efficacy of large-scale search and rescue missions. In this work, we leverage the advantages of drone-based vision for RGBT person detection. The drone-based RGBT person detection task brings interesting challenges to existing cross-modality object detectors, e.g., tiny sizes of objects, modality-space imbalance, and position shifts. Observing that there is a lack of data and customized detectors for drone-based RGBT person detection, we contribute two new datasets and design a novel detector. The data contribution is two-fold. For one, we construct the first large-scale drone-based RGBT person detection benchmark RGBTDronePerson, which contains 6,125 pairs of RGBT images and 70,880 instances. Images are captured in various scenes and under various illumination and weather conditions. For another, we annotate the VTUAV tracking dataset and obtain its object detection version, named VTUAV-det. To tackle the challenges raised by this task, we propose a Quality-aware RGBT Fusion Detector (QFDet). Firstly, we design a Quality-aware Learning Strategy (QLS) to provide sufficient supervision for tiny objects while focusing on high-quality samples, in which a Quality-Aware Factor (QAF) is designed to measure the quality. Moreover, a Quality-aware Cross-modality Enhancement module (QCE) is proposed to predict a QAF map for each modality, which not only indicates the reliability of each modality but also highlights regions where objects are more likely to appear. Our QFDet remarkably boosts the detection performance over tiny and small objects, surpassing the strong baseline on mAPtiny 50 by 6.57 points on RGBTDronePerson and mAPs by 3.80 points on VTUAV-det. The datasets, codes, and pre-trained models are available at https://nnnnerd.github.io/RGBTDronePerson/.