Chart reviews in emergency medicine research: Where are the methods?

Study objective: Medical chart reviews are often used in emergency medicine research. However, the reliability of data abstracted by chart reviews is seldom examined critically The objective of this investigation was to determine the proportion of emergency medicine research articles that use data from chart reviews and the proportions that report methods of case selection, abstractor training, monitoring and blinding, and interrater agreement. Methods: Research articles published in three emergency medicine journals from January 1989 through December 1993 were identified. The articles that used chart reviews were analyzed. Results: Of 986 original research articles that were identified, 244 (25%; 95% confidence interval [CI], 22% to 28%) relied on chart reviews. Inclusion criteria were described in 98% (95% Cl, 96% to 99%), and 73% (95% Cl, 67% to 79%) defined the variables being analyzed. Other methods were seldom mentioned: abstractor training, 18% (95% Cl, 13% to 23%); standardized abstraction forms, 11% (95% Cl, 7% to 15%); periodic abstractor monitoring, 4% (95% Cl, 2% to 7%); and abstractor blinding to study hypotheses, 3% (95% Cl, 1% to 6%). Interrater reliability was mentioned in 5% (95% Cl, 3% to 9%) and tested statistically in .4% (95% Cl, 0% to 2%). A 15% random sample of articles was reassessed by a second investigator; interrater agreement was high for all eight criteria. Conclusion: Chart review is a common method of data collection in emergency medicine research. Yet, information about the quality of the data is usually lacking. Chart reviews should be held to higher methodologic standards, or the conclusions of these studies may be in error.