Continuously Cumulating Meta-Analysis and Replicability

被引:260
作者
Braver, Sanford L. [1 ,2 ]
Thoemmes, Felix J. [3 ]
Rosenthal, Robert [2 ]
机构
[1] Arizona State Univ, Tempe, AZ 85287 USA
[2] Univ Calif Riverside, Riverside, CA 92521 USA
[3] Cornell Univ, Ithaca, NY 14853 USA
关键词
effect-size heterogeneity; statistical intuition; meta-analysis; replication; PSYCHOLOGICAL-RESEARCH; STATISTICAL POWER; HETEROGENEITY; REPLICATION; BIAS; INTERVALS; TRUTH; SIZE;
D O I
10.1177/1745691614529796
中图分类号
B84 [心理学];
学科分类号
04 ; 0402 ;
摘要
The current crisis in scientific psychology about whether our findings are irreproducible was presaged years ago by Tversky and Kahneman (1971), who noted that even sophisticated researchers believe in the fallacious Law of Small Numbers-erroneous intuitions about how imprecisely sample data reflect population phenomena. Combined with the low power of most current work, this often leads to the use of misleading criteria about whether an effect has replicated. Rosenthal (1990) suggested more appropriate criteria, here labeled the continuously cumulating meta-analytic (CCMA) approach. For example, a CCMA analysis on a replication attempt that does not reach significance might nonetheless provide more, not less, evidence that the effect is real. Alternatively, measures of heterogeneity might show that two studies that differ in whether they are significant might have only trivially different effect sizes. We present a nontechnical introduction to the CCMA framework (referencing relevant software), and then explain how it can be used to address aspects of replicability or more generally to assess quantitative evidence from numerous studies. We then present some examples and simulation results using the CCMA approach that show how the combination of evidence can yield improved results over the consideration of single studies.
引用
收藏
页码:333 / 342
页数:10
相关论文
共 66 条
[1]   An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science [J].
Alexander, Anita ;
Barnett-Cowan, Michael ;
Bartmess, Elizabeth ;
Bosco, Frank A. ;
Brandt, Mark ;
Carp, Joshua ;
Chandler, Jesse J. ;
Clay, Russ ;
Cleary, Hayley ;
Cohn, Michael ;
Costantini, Giulio ;
DeCoster, Jamie ;
Dunn, Elizabeth ;
Eggleston, Casey ;
Estel, Vivien ;
Farach, Frank J. ;
Feather, Jenelle ;
Fiedler, Susann ;
Field, James G. ;
Foster, Joshua D. ;
Frank, Michael ;
Frazier, Rebecca S. ;
Fuchs, Heather M. ;
Galak, Jeff ;
Galliani, Elisa Maria ;
Garcia, Sara ;
Giammanco, Elise M. ;
Gilbert, Elizabeth A. ;
Giner-Sorolla, Roger ;
Goellner, Lars ;
Goh, Jin X. ;
Goss, R. Justin ;
Graham, Jesse ;
Grange, James A. ;
Gray, Jeremy R. ;
Gripshover, Sarah ;
Hartshorne, Joshua ;
Hayes, Timothy B. ;
Jahn, Georg ;
Johnson, Kate ;
Johnston, William ;
Joy-Gaba, Jennifer A. ;
Lai, Calvin K. ;
Lakens, Daniel ;
Lane, Kristin ;
LeBel, Etienne P. ;
Lee, Minha ;
Lemm, Kristi ;
Mackinnon, Sean ;
May, Michael .
PERSPECTIVES ON PSYCHOLOGICAL SCIENCE, 2012, 7 (06) :657-660
[2]   Research in the psychological laboratory: Truth or triviality? [J].
Anderson, CA ;
Lindsay, AJ ;
Bushman, BJ .
CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE, 1999, 8 (01) :3-9
[3]  
[Anonymous], P CURVE FIXES PUBLIC
[4]  
[Anonymous], 2021, INTRO META ANAL
[5]  
[Anonymous], 2008, PRACTICAL METAANALYS
[6]  
[Anonymous], INVESTIGATING VARIAT, DOI DOI 10.1027/1864-9335/A000178
[7]  
[Anonymous], PROFESSORS ARE NOT E
[8]  
[Anonymous], 1988, Statistical Power Analysis for the Behavioral Sciences
[9]  
[Anonymous], 2013, UNDERSTANDING NEW ST
[10]  
[Anonymous], 2012, CHRONICLE HIGHER ED