Multi-Armed RCTs: A Design-Based Framework. NCEE 2017-4027

Autor/in	Schochet, Peter Z.
Institution	National Center for Education Evaluation and Regional Assistance (ED); Decision Information Resources, Inc.
Titel	Multi-Armed RCTs: A Design-Based Framework. NCEE 2017-4027
Quelle	(2017), (36 Seiten) PDF als Volltext (1); PDF als Volltext kostenfreie Datei (2) Verfügbarkeit
Zusatzinformation	Weitere Informationen
Sprache	englisch
Dokumenttyp	gedruckt; online; Monographie
Schlagwörter	Design; Randomized Controlled Trials; Quasiexperimental Design; Research Methodology; Educational Research; Intervention; Measures (Individuals); Equations (Mathematics); Computation; Program Evaluation; Program Effectiveness; Evaluation Methods; Experiments; Models; Regression (Statistics); Hierarchical Linear Modeling; Least Squares Statistics + Suchen Sie Ihr Suchwort? Research method; Forschungsmethode; Bildungsforschung; Pädagogische Forschung; Messdaten; Equations; Mathematics; Gleichungslehre; Programme evaluation; Programmevaluation; Erprobung; Analogiemodell; Regression; Regressionsanalyse
Abstract	Design-based methods have recently been developed as a way to analyze data from impact evaluations of interventions, programs, and policies (Imbens and Rubin, 2015; Schochet, 2015, 2016). The estimators are derived using the building blocks of experimental designs with minimal assumptions, and are unbiased and normally distributed in large samples with simple variance estimators. The methods apply to randomized controlled trials (RCTs) and quasi-experimental designs (QEDs) with comparison groups for a wide range of designs used in social policy research. The methods have important advantages over traditional model-based impact estimation methods, such as hierarchical linear model (HLM) and robust cluster standard error (RCSE) methods, and perform well in simulations (Schochet, 2016). The free "RCT-YES" software (www.rct-yes.com) estimates and reports impacts using these design-based methods. This report discusses several key topics for estimating average treatment effects (ATEs) for multi-armed designs. The report is geared toward methodologists with a strong background in statistical theory and a good knowledge of design-based concepts for the single treatment-control group (two-group) design. The report builds on Schochet (2016), referencing key results and formulas to avoid repetition, and serves as a supplement to that report. The focus is on RCTs, although key concepts apply also to QEDs with comparison groups. The report is in three sections. Section 1 discusses how design-based ATE estimators for the two-group design need to be modified for the multi-armed design when comparing pairs of research groups to each other. Section 2 discusses multiple comparison adjustments when conducting hypothesis tests across pairwise contrasts to identify the most effective interventions. Finally, Section 3 shows that the assumptions required to identify and estimate the complier average causal effect (CACE) parameter using an instrumental variable (IV) framework become much more complex in the multi-armed context, and may not be possible in some cases. While Sections 2 and 3 are germane to multi-armed designs regardless of the impact estimation methods used for the analysis, these sections emphasize approaches that align with the non-parametric underpinnings of the design-based framework. [For related reports see: "Multi-Armed RCTs: A Design-Based Framework. NCEE 2017-4027" (ED575014) and "Comparing Impact Findings from Design-Based and Model-Based Methods: An Empirical Investigation. NCEE 2017-4026" (ED575021).] (ERIC).
Anmerkungen	National Center for Education Evaluation and Regional Assistance. Available from: ED Pubs. P.O. Box 1398, Jessup, MD 20794-1398. Tel: 877-433-7827; Web site: http://ies.ed.gov/ncee/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01