Critical thinking in graduate medical education - A role for concept mapping assessment?

Context Tools to assess the evolving conceptual framework of physicians-in-training are limited, despite their critical importance to physicians' evolving clinical expertise. Concept mapping assessment (CMA) enables teachers to view students' organization of their knowledge at various points in training. Objective To assess whether CMA reflects expected differences and changes in the conceptual framework of resident physicians, whether concept maps can be scored reliably, and how well CMA scores relate to the results of standard in-training examination. Design, Setting, and Participants A group of 21 resident physicians (9 first-year and 12 second- and third-year residents) from a university-based pediatric training program underwent concept map training, drew a preinstruction concept map about seizures, completed an education course on seizures, and then drew a postinstruction map. Maps were scored independently by 3 raters using a standardized method. The study was conducted in May and June 1999. Main Outcome Measures Preinstruction map total scores and subscores in 4 categories compared with postinstruction map scores; map scores of second- and third-year residents compared with first-year residents; and interrater correlation of map scores. Results Total CMA scores increased after instruction from a mean (SD) preinstruction map score of 429 (119) to a mean postinstruction map score of 516 (196) (P=.03). Second- and third-year residents scored significantly higher than first year residents before instruction (mean [SD] score of 472 [116] vs 371 [102], respectively; P=.04), but not after instruction (mean [SD] scores, 561 [203] vs 456 [179], respectively; P=.16). Second- and third-year residents had greater preinstruction map complexity as measured by cross-link score (P=.01) than first-year residents. The CMA score had a weak to no correlation with the American Board of Pediatrics In-training Examination score (r=0.10-0.54). Interrater correlation of map scoring ranged from weak to moderate for the preinstruction map (r=0.51-0.69) and moderate to strong for the postinstruction map (r=0.74-0.88). Conclusions Our data provide preliminary evidence that concept mapping assessment reflects expected differences and change in the conceptual framework of resident physicians. Concept mapping assessment and standardized testing may measure different cognitive domains.