Active Learning for Testing and Evaluation in Field Robotics: A Case Study in Autonomous, Off-Road Navigation

Testing and evaluation of field robotic systems requires both experimentation in representative conditions and human supervision to effectively assess components, manage risk, and interpret results. Due to the complexity of robotic systems, we argue this experimentation should be done adaptively by using insights gained from previous trials. Furthermore, we envision an advisory system that could assist experimenters with selecting trial configurations by learning and accounting for human preferences and risk tolerances; however, formal methods for human decision making in the context of field robotic experimentation remains an open question. In this work, we present and analyze a case study for how decisions were made during the testing and evaluation of an off-road, autonomous navigation system. From the perspective of active learning, we find that Bayesian Optimization is a promising mathematical framework for modeling human decision making in adaptive experimental design of field robotics and that a combination of the EI, KG, and PES acquisition functions would likely be useful for realizing an advisory system.