Reliability and validity of performance measures in microworlds

Research with microworlds enable us to study complex mental processes in semantically rich but standardized environments, under more realistic conditions than ordinary tests do. Moreover, adapting the microworlds to the needs of mental testing may increase our ability to predict success in real life pursuits [Rigas, G., & Brehmer, B. (1999). Mental processes in intelligence tests and dynamic decision making tasks. In P. Juslin & H. Montgomery (Eds.), Judgement and decision making: Neo-Brunswikean and process-tracing approaches (pp. 45-65). Hillsdale, NJ: Lawrence Erlbaum Associates.]. Yet earlier studies have found no relations between intelligence test scores and microworld performance. The relationships between two real time dynamic decision making tasks (microworlds) and one intelligence test were investigated. The participants interacted with eight NEWFIRE scenarios, four COLDSTORE scenarios, and completed Raven's Advanced Progressive Matrices (APM). The composite scores of performance in the two microworlds were sufficiently reliable. APM was a predictor of success in these two microworlds. Statistically significant increases in R 2 were obtained. Methodological problems in research with microworlds are discussed and changes related to the adaptation of microworlds to the needs of mental testing are proposed. (C) 2002 Elsevier Science Inc. All rights reserved.