Causal Inference and the Comparative Interrupted Time Series Design: Findings from Within-Study Comparisons

Autor/inn/en	St. Clair, Travis; Hallberg, Kelly; Cook, Thomas D.
Institution	Society for Research on Educational Effectiveness (SREE)
Titel	Causal Inference and the Comparative Interrupted Time Series Design: Findings from Within-Study Comparisons
Quelle	(2014), (14 Seiten) PDF als Volltext kostenfreie Datei Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Monographie
Schlagwörter	Measurement Techniques; Time; Randomized Controlled Trials; Comparative Analysis; Performance; Benchmarking; Grade 4; Grade 5; Grade 6; Scores; Achievement Tests; English; Language Arts; Mathematics; Florida; Indiana Statewide Testing for Educational Progress Plus + Suchen Sie Ihr Suchwort? Messtechnik; Zeit; Achievement; Leistung; School year 04; 4. Schuljahr; Schuljahr 04; School year 05; 5. Schuljahr; Schuljahr 05; School year 06; 6. Schuljahr; Schuljahr 06; Achievement test; Testing; Test; Tests; Leistungsbeurteilung; Leistungsüberprüfung; Testdurchführung; Testen; English language; Englisch; Sprachkultur; Mathematik
Abstract	Researchers are increasingly using comparative interrupted time series (CITS) designs to estimate the effects of programs and policies when randomized controlled trials are not feasible. In a simple interrupted time series design, researchers compare the pre-treatment values of a treatment group time series to post-treatment values in order to assess the impact of a treatment, without any comparison group to account for confounding factors. The CITS design is a version of the ITS design in which both a treatment and a comparison group are evaluated both before and after the onset of a treatment. A growing body of literature is employing a within study comparison (WSC) methodology to examine the validity of the CITS model. WSC studies empirically estimate the extent to which a given observational study reproduces the results of a randomized controlled trial (RCT) when both share the same treatment group, and represent a rigorous method of evaluating non-experimental designs using real data. A number of recent within-study comparisons have demonstrated that CITS can produce estimates that are comparable to those from a randomized controlled trial (RCT) in practice. In the St. Clair et al. (2014) application, the authors found that correspondence with the RCT was possible when the CITS model accounted for baseline trends, but that additional time points could actually increase bias when the pre-treatment trend was not modeled correctly. Examination of the pretreatment trends in this data set showed clearly that in at least one of the outcomes the treatment and comparison groups had different slopes in the pretreatment period, and as a result the "parallel trends" assumption often invoked in the difference-in-difference literature was clearly violated. This paper employs a within study comparison (WSC) methodology to examine the performance of two approaches: (1) a more flexible modeling approach, which employs year fixed-effects rather than trying to parametrically model the pretest trend; and (2) match treatment and comparison cases to reduce reliance on modeling the pretreatment trend. The paper then compares the approaches to the performance of the baseline mean and baseline slope models across three datasets. The purpose of this research is two-fold: (1) to examine what approach, if any, works in the unclear functional form case; and (2) to examine the relative superiority of the different approaches across the three datasets in terms of both bias reduction and precision. Tables and figures are appended. (ERIC).
Anmerkungen	Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries@sree.org; Web site: http://www.sree.org
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01