Modeling Item-Level Heterogeneous Treatment Effects with the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions

Autor/inn/en	Gilbert, Joshua B.; Kim, James S.; Miratrix, Luke W.
Titel	Modeling Item-Level Heterogeneous Treatment Effects with the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions
Quelle	In: Journal of Educational and Behavioral Statistics, 48 (2023) 6, S.889-913 (25 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Gilbert, Joshua B.) ORCID (Kim, James S.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1076-9986
DOI	10.3102/10769986231171710
Schlagwörter	Test Items; Item Response Theory; Computer Assisted Testing; Program Effectiveness; Intervention; Outcome Measures; Error of Measurement; Literacy Education; Grade 3; Elementary School Students; Reading Comprehension + Suchen Sie Ihr Suchwort? Test content; Testaufgabe; Item-Response-Theorie; Messfehler; School year 03; 3. Schuljahr; Schuljahr 03; Leseverstehen
Abstract	Analyses that reveal how treatment effects vary allow researchers, practitioners, and policymakers to better understand the efficacy of educational interventions. In practice, however, standard statistical methods for addressing heterogeneous treatment effects (HTE) fail to address the HTE that may exist "within" outcome measures. In this study, we present a novel application of the explanatory item response model (EIRM) for assessing what we term "item-level" HTE (IL-HTE), in which a unique treatment effect is estimated for each item in an assessment. Results from data simulation reveal that when IL-HTE is present but ignored in the model, standard errors can be underestimated and false positive rates can increase. We then apply the EIRM to assess the impact of a literacy intervention focused on promoting transfer in reading comprehension on a digital assessment delivered online to approximately 8,000 third-grade students. We demonstrate that allowing for IL-HTE can reveal treatment effects at the item-level masked by a null average treatment effect, and the EIRM can thus provide fine-grained information for researchers and policymakers on the potentially heterogeneous causal effects of educational interventions. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01