Measuring Difficulty of Learning Using Ensemble Methods

Measuring the difficulty of each instance is a crucial metaknowledge extraction problem. Most studies on data complexity have focused on extracting the characteristics at a dataset level instead of the instance level while also requiring the complete label knowledge of the dataset, which can often be expensive to obtain. At the instance level, the most commonly used metrics to determine difficult to classify instances are dependant on the learning algorithm used (i.e., uncertainty), and are measurements of the entire system instead of only the dataset. Additionally, these metrics only provide information of misclassification in regard to the learning algorithm and not in respect of the composition of the instances within the dataset. We introduce and propose several novel instance difficulty measures in a semi-supervised boosted ensemble setting to identify difficult to classify instances based on their learning difficulty in relation to other instances within the dataset. The proposed difficulty measures measure both the fluctuations in labeling during the construction process of the ensemble and the amount of resources required for the correct label. This provides the degree of difficulty and gives further insight into the origin of classification difficulty at the instance level reflected by the scores of different difficulty measures.