Testing Object Detection Models For Autonomous Vehicles Against Hazards

Perception systems play a crucial role in ensuring the safety of autonomous vehicles. These systems must be capable of navigating a variety of challenging conditions that could occur during autonomous driving, including natural hazards such as heavy precipitation or raindrops obscuring the camera lens. As a result, it is of utmost importance that these perception systems undergo rigorous testing against these potential hazards, a requirement enforced by many countries' regulatory bodies for human-operated vehicles. However, given the multitude of potential hazard scenarios, each with its own set of adjustable parameters, two significant challenges arise. The first challenge is to devise a systematic and comprehensive method for testing autonomous vehicles against these hazard scenarios, with quantifiable results. The second challenge involves efficiently navigating the vast search space to pinpoint scenarios that could potentially lead to system failure. In our research, we present HazardTesting, a framework for generating a diverse, customizable hazard catalogue for autonomous vehicle perception evaluation. This tool not only measures an AV's hazard testing coverage but also identifies critical adversarial hazards via optimization procedures. Encompassing 70 unique hazards relevant to AV visual perception, HazardTesting employs a Genetic Algorithm for efficient parameter optimization, meeting different testing objectives. Implemented in Unity, a leading 3D engine, it allows comprehensive AV hazard testing. Through extensive experiments on YOLO and Faster RCNN, two prevalent AV perception models, we found substantial room for improvement. Notably, our optimization-based testing outperformed random methods by detecting 182.2% more perceptual errors on average.