Predictive Modeling of Rater Behavior: Implications for Quality Assurance in Essay Scoring

Autor/inn/en	Bejar, Isaac I.; Li, Chen; McCaffrey, Daniel
Titel	Predictive Modeling of Rater Behavior: Implications for Quality Assurance in Essay Scoring
Quelle	In: Applied Measurement in Education, 33 (2020) 3, S.234-247 (14 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0895-7347
DOI	10.1080/08957347.2020.1750406
Schlagwörter	Scoring; Essays; Behavior; Predictive Measurement; Predictor Variables; Models; Individual Differences; Quality Control; Essay Tests; Evaluators + Suchen Sie Ihr Suchwort? Bewertung; Essay; Aufsatzunterricht; Prädiktor; Analogiemodell; Individueller Unterschied; Qualitätskontrolle; Schriftlicher Sprachgebrauch
Abstract	We evaluate the feasibility of developing predictive models of rater behavior, that is, "rater-specific" models for predicting the scores produced by a rater under operational conditions. In the present study, the dependent variable is the score assigned to essays by a rater, and the predictors are linguistic attributes of the essays used by the e-rater® engine. Specifically, for each rater, the linear regression of rater scores on the linguistic attributes is obtained based on data from two consecutive time periods. The regression from each period was cross validated against data from the other period. Raters were characterized in terms of their level of predictability and the importance of the predictors. Results suggest that rater models capture stable individual differences among raters. To evaluate the feasibility of using rater models as a quality control mechanism, we evaluated the relationship between rater predictability and inter-rater agreement and performance on pre-scored essays. Finally, we conducted a simulation whereby raters are simulated to score exclusively as a function of essay length at different points during the scoring day. We concluded that predictive rater models merit further investigation as a means of quality controlling human scoring. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01