"How Do Raters Learn to Rate?" Many-Facet Rasch Modeling of Rater Performance over the Course of a Rater Certification Program

Autor/inn/en	Yan, Xun; Chuang, Ping-Lin
Titel	"How Do Raters Learn to Rate?" Many-Facet Rasch Modeling of Rater Performance over the Course of a Rater Certification Program
Quelle	In: Language Testing, 40 (2023) 1, S.153-179 (27 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Yan, Xun) ORCID (Chuang, Ping-Lin)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0265-5322
DOI	10.1177/02655322221074913
Schlagwörter	Evaluators; Interrater Reliability; Item Response Theory; Certification; Training; Second Language Learning; English (Second Language); Writing Tests; Placement Tests; Essay Tests; Scaling; Learning Processes; Program Effectiveness; Writing Evaluation + Suchen Sie Ihr Suchwort? Interrater-Reliabilität; Item-Response-Theorie; Abschlusszeugnis; Zertifizierung; Ausbildung; Zweitsprachenerwerb; English as second language; English; Second Language; Englisch als Zweitsprache; Writing test; Schreibtest; Schriftlicher Sprachgebrauch; Scale construction; Skalenkonstruktion; Learning process; Lernprozess
Abstract	This study employed a mixed-methods approach to examine how rater performance develops during a semester-long rater certification program for an English as a Second Language (ESL) writing placement test at a large US university. From 2016 to 2018, we tracked three groups of novice raters (n = 30) across four rounds in the certification program. Using many-facet Rasch modeling, rater performance was examined in terms of rater agreement, rater consistency, and rater severity. These measurement estimates of rating quality were subjected to multivariate analysis to examine whether and how rater performance changes across rounds. Rater comments on the essays were qualitatively analyzed to obtain a deeper understanding of how raters learn to use the scale over time. The quantitative results showed a non-linear, three-staged developmental pattern of rater performance for all three groups of raters. Findings of this study suggest that rater development resembles a learning curve similar to how one acquires a language and other skills. We argue that understanding the developmental pattern of rater behavior is crucial not only to understanding the effectiveness of rater training, but also to the investigation of rater cognition and development. We will also discuss the practical implications of this study in relation to the effort and expectations needed for rater training for writing assessments. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01