Prognostic Score-Based Difference-in-Differences Strategy for Multilevel Multi-Cohort Data

Autor/in	Hong, Guanglei
Institution	Society for Research on Educational Effectiveness (SREE)
Titel	Prognostic Score-Based Difference-in-Differences Strategy for Multilevel Multi-Cohort Data
Quelle	(2013), (8 Seiten) PDF als Volltext kostenfreie Datei Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Monographie
Schlagwörter	Accountability; Scores; Comparative Analysis; Algebra; Grade 9; Mathematics Achievement; Secondary School Mathematics; Experimental Groups; Academic Achievement; Statistical Analysis; Measurement; Bias; Prognostic Tests; Cohort Analysis; Illinois + Suchen Sie Ihr Suchwort? Verantwortung; School year 09; 9. Schuljahr; Schuljahr 09; Mathmatics sikills; Mathmatics achievement; Mathematical ability; Mathematische Kompetenz; Schulleistung; Statistische Analyse; Messverfahren; Prognostischer Test; Kohortenanalyse
Abstract	When using time series accountability data to evaluate system-wide education policies, concurrent changes often pose threats to internal validity. The standard difference-in-differences (DID) method resorts to a non-equivalent comparison group whose average outcome change is due to such confounding. This strategy relies on the strong assumption that the average confounding impact of concurrent events is the same for the comparison group unaffected by the policy and the experimental group affected by the policy. This assumption will be violated and therefore the DID results will be biased, for example, if the confounding effect varies by individual characteristics and if the experimental group and the comparison group differ in such characteristics. Prior research has typically employed a DID model with linear covariance adjustment for observed pretreatment characteristics. More recently, researchers have attempted to equate the covariate distribution of the comparison group with that of the experimental group through propensity score matching or weighting before conducting DID analyses. This study provides the theoretical rationale for the prognostic score-based DID strategy, clarifies its identification assumptions, and develops an analytic procedure. This study illustrates the new strategy with an evaluation of a policy adopted by the Chicago Public Schools requiring all ninth graders to take algebra, defines the causal estimand, and develops statistical models for investigating whether the policy effect was enhanced as its implementation became mature or whether the effect faded out over time as the reform lost its momentum after the initial period. The new strategy is then extended to multilevel multi-cohort education accountability data. This new strategy greatly reduces the dimensionality of covariates for adjustment, which is a major advantage over the linear covariance adjusted DID. The stratification procedure enables researchers to detect heterogeneity in the confounding effects of concurrent events as well as in the policy effect. Yet like most other DID methods, this new strategy shows limitations when the experimental group and the conditional group differ in the distribution of an unobservable and when the amount of confounding of concurrent events is a function of the unobservable. Appendices include a proof and information on obtaining an estimate of the policy effect on the untreated pre-policy students in the experimental school. (ERIC).
Anmerkungen	Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; Fax: 202-640-4401; e-mail: inquiries@sree.org; Web site: http://www.sree.org
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01