Assessing the validity of multiple-choice questions for RAPP story comprehension

Background: Passage dependency (PD) is a measure of how much information is required to answer test questions based only on information provided in the relevant text (Tuiman, 1974). Prior learning, information included in other test questions or responses, and the ability to eliminate less plausible or irrelevant foils are all factors that may affect PD. The PD of the multiple-choice questions for the auditory story comprehension task in the RAPP software environment has yet to be established. Aims: The purpose of this experiment was to investigate the validity of newly developed multiple-choice comprehension questions for the story comprehension tasks used in the RAPP software environment. Methods & Procedures: Participants were 40 young adults without speech or language impairment, 20 of whom heard each of 12 stimulus stories and answered 10 multiple-choice questions with 5 response choices, and 20 who answered the multiple-choice questions but were not exposed to the stimulus stories. Questions concerned information about stated and implied main ideas and details presented in the stories. Based on four pre-determined story forms (three stories each), analyses examined the questions' validity in terms of their PD compared to chance performance and to a pre-established Passage Dependency Index (PDI) criterion (.60). Significant differences in the PDI and in the percentage of correctly answered questions among predetermined forms were also evaluated. Analyses also examined the PD of questions based on the type of information queried both within and among forms. Outcomes & Results: Results provide support for the validity of the multiple-choice questions without consideration of the nature of the information queried by them. The percentage of correctly answered questions exceeded chance for all story forms, PDIs met or exceeded the pre-established criterion, there were no significant differences in the PDIs among the four forms and there were no significant differences in the percentage of correctly answered questions among the four forms. There were no significant differences in the PDIs among the four question types by story form or among question types within a form and all PDIs met or exceeded pre-established criteria (except detail implied questions in one form). Conclusions: These results provide support for the PD of the multiple-choice questions constructed for the four story forms that make up this version of a story-length auditory comprehension task. Results suggest that the questions are not biased relative to the content of the stories and results add to the overall validity of the task.