Quantifying "Promising Trials Bias" in Randomized Controlled Trials in Education

Autor/inn/en	Sims, Sam; Anders, Jake; Inglis, Matthew; Lortie-Forgues, Hugues
Titel	Quantifying "Promising Trials Bias" in Randomized Controlled Trials in Education
Quelle	In: Journal of Research on Educational Effectiveness, 16 (2023) 4, S.663-680 (18 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Sims, Sam) ORCID (Anders, Jake) ORCID (Inglis, Matthew) ORCID (Lortie-Forgues, Hugues)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1934-5747
DOI	10.1080/19345747.2022.2090470
Schlagwörter	Randomized Controlled Trials; Educational Research; Effect Size; Intervention; Statistical Bias; Research Problems; Statistical Analysis; Statistical Significance + Suchen Sie Ihr Suchwort? Bildungsforschung; Pädagogische Forschung; Forschungskritik; Statistische Analyse
Abstract	Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results (p < [alpha]) We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01