Learning Algorithm in Two-Stage Selective Prediction

Data gathered from real-world applications often suffer from corruption. The low-quality data will hinder the performance of the learning system in terms of classification accuracy, model building time, and interpretability of the classifier. Selective prediction, also known as prediction with a reject option, is to reduce the error rate by abstaining from prediction under uncertainty while keeping coverage as high as possible. Deep Neural Network (DNN) has a high capacity for fitting large-scale data. If DNNs can leverage the tradeoff coverage by selective prediction, then the performance can potentially be improved. However, the current DNN embedded with the reject option requires the knowledge of the rejection threshold, and the searching of threshold is inefficient in large-scale applications. Besides, the abstention of prediction on partial datasets increases the model bias and might not be optimal. To resolve these problems, we propose innovative threshold learning algorithms integrated with the selective prediction that can estimate the intrinsic rejection rate of the dataset. Correspondingly, we provide a rigorous framework to generalize the estimation of data corruption rate. To leverage the advantage of multiple learning algorithms, we extend our learning algorithms to a hierarchical two-stage system. Our methods have the advantage of being flexible with any neural network architecture. The empirical results show that our algorithms can achieve state-of-the-art performance in challenging real-world datasets in both classification and regression problems.