Using Linkage Sets to Improve Connectedness in Rater Response Model Estimation

Autor/inn/en	Casabianca, Jodi M.; Donoghue, John R.; Shin, Hyo Jeong; Chao, Szu-Fu; Choi, Ikkyu
Titel	Using Linkage Sets to Improve Connectedness in Rater Response Model Estimation
Quelle	In: Journal of Educational Measurement, 60 (2023) 3, S.428-454 (27 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Shin, Hyo Jeong)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0022-0655
DOI	10.1111/jedm.12360
Schlagwörter	Item Response Theory; Alternative Assessment; Evaluators; Research Problems; Scoring; Scores; Computer Assisted Testing; Test Scoring Machines; Automation; Design + Suchen Sie Ihr Suchwort? Item-Response-Theorie; Forschungskritik; Bewertung
Abstract	Using item-response theory to model rater effects provides an alternative solution for rater monitoring and diagnosis, compared to using standard performance metrics. In order to fit such models, the ratings data must be sufficiently connected in order to estimate rater effects. Due to popular rating designs used in large-scale testing scenarios, there tends to be a large proportion of missing data, yielding sparse matrices and estimation issues. In this article, we explore the impact of different types of connectedness, or linkage, brought about by using a linkage set--a collection of responses scored by most or all raters. We also explore the impact of the properties and composition of the linkage set, the different connectedness yielded from different rating designs, and the role of scores from automated scoring engines. In designing monitoring systems using the rater response version of the generalized partial credit model, the study results suggest use of a linkage set, especially a large one that is comprised of responses representing the full score scale. Results also show that a double-human-scoring design provides more connectedness than a design with one human and an automated scoring engine. Furthermore, scores from automated scoring engines do not provide adequate connectedness. We discuss considerations for operational implementation and further study. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01