Accuracy of Effect Size Estimates From Published Psychological Experiments Involving Multiple Trials

The reporting of exaggerated effect size estimates may occur either through researchers accepting statistically significant results when power is inadequate and/or from repeated measures approaches aggregating, averaging multiple items, or multiple trials. Monte-Carlo simulations with input of a small, medium, or large effect size were conducted on multiple items or trials that were either averaged or aggregated to create a single dependent measure. Alpha was set at the .05 level, and the trials were assessed over item or trial correlations ranging from 0 to 1. Simulations showed a large increase in observed effect size averages and the power to accept these estimates as statistically significant increased over numbers of trials or items. Overestimation effects were mitigated as correlations between trials increased but still remained substantial in some cases. The implications of these findings for meta-analyses and different research scenarios are discussed.