Accelerated Evaluation of Autonomous Drivers using Neural Network Quantile Generators

Whereas autonomous driver engineering has limited control on outcomes for individual scenarios, engineering processes must exert control over the performance statistics. A key challenge impeding statistical confidence is the need to test for a long list of infrequent hazardous events. Using "smart miles" promises to reduce the evaluation cost by increasing the frequency of those infrequent hazardous events. We propose a simulation based Bayesian approach which increases the frequency of those infrequent events by many orders of magnitudes as compared to naturalistic miles. We represent the Operational Design Domain (ODD) using a population of scenarios, and provide methods for sampling the ODD. We propose a quantile function based sampling approach which is able to generate a single sample with thousands of instances using a single forward pass of a deep neural network (DNN). We develop a practical sampler training method for an "inverted DNN" architecture. The resulting sampler is capable of generating a skewed distribution comprising of "smart miles" in which hazardous events of interest occur at a frequency >95%. To gauge the quality of the generated sample we propose the quality metrics of histogram diversity, histogram homoscedasticity and average sample distance.