Who's the Best? A Relativistic View of Expertise

The dictionary and the expert performance approach view an expert as one who, after sufficient training and experience in a domain, can perform the requisite tasks above a threshold level. In contrast, we argue for a performance-based approach that implies expertise is a continuum; the experts are the best performers. Most tasks in which expertise can be demonstrated have an underlying core of judgment, including domains in which the tasks call for judgment to be overlain with performance, prediction, or instruction. To evaluate judgment, we employ the metaphor of the judge as a measuring instrument. Like an instrument, expert judgment according to the performance-based approach has three key properties: discrimination, consistency, and validity. Validity requires ground truth and is usually difficult to establish; but the other two properties are readily observable, and they are combined in the Cochran-Weiss-Shanteau index. Copyright (C) 2014 John Wiley & Sons, Ltd.