Test and Evaluation of Autonomous Surface Vehicles: A Case Study

The rigorous test and evaluation of autonomous systems is often a challenging process. Specifically, the transformation of qualitative assessments often used in the development phase to unbiased quantitative assessment can be difficult. Additionally, few standardized metrics exist for the test and evaluation of autonomous systems, as applications/approaches vary so widely that specialized metrics must be developed in the majority of cases. This paper presents a case study for this process, describing a multi-year, iterative test and evaluation effort undertaken by the authors in support of the development of an autonomous surface vehicle system. The case study details the development of quantitative performance metrics (including the rationale behind each one), revisions/additions that were made to the metrics between testing and evaluation events, impacts the metrics had on the autonomy development process observed through successive tests, and lessons learned throughout the process.