Towards detection of learner misconceptions in a medical learning environment: a subgroup discovery approach

Ill-structured problems, by definition, have multiple paths to a solution and are multifaceted making automated assessment and feedback a difficult challenge. Diagnostic reasoning about medical cases meet the criteria of ill-structured problem solving since there are multiple solution paths. The goal of this study was to develop an adaptive feedback mechanism that is capable of identifying and responding to novice physician misconceptions by mining the log trace data of user interactions in BioWorld, a computer-based learning environment designed to support medical students in regulating their own diagnostic reasoning. We applied a series of text pre-processing algorithms to extract the linguistic features that characterized symptoms identified by 30 medical students solving three endocrinology cases: diabetes mellitus (type 1), Pheochromocytoma, and Hyperthyroidism. A subgroup discovery mining algorithm was applied in two steps. First, rules were induced to predict an incorrect diagnosis submitted as the final solution for a case on the basis of symptoms highlighted by medical students as being pertinent, that were in fact not pertinent. Second, rules were induced to predict a distractor hypothesis (an incorrect hypothesis listed as the most probable) during the differential diagnosis process while solving the case. The rule set discovered through the subgroup discovery task was shown to predict both incorrect and distractor hypotheses set by novice physicians while solving the case. We discuss the implications in terms of developing an adaptive feedback mechanism that can detect physicians' misconceptions and errors during problem-solving as a means to deliver prompts and feedback that promote the acquisition of metacognitive monitoring and control skills.