Beyond Agreement: Exploring Rater Effects in Large-Scale Mixed Format Assessments

Autor/inn/en	Wind, Stefanie A.; Guo, Wenjing
Titel	Beyond Agreement: Exploring Rater Effects in Large-Scale Mixed Format Assessments
Quelle	In: Educational Assessment, 26 (2021) 4, S.264-283 (20 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Wind, Stefanie A.) ORCID (Guo, Wenjing)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1062-7197
DOI	10.1080/10627197.2021.1962277
Schlagwörter	Scoring; Responses; Test Items; Test Format; National Competency Tests; Evaluators; Evaluative Thinking; Educational Assessment; Evaluation Methods; National Assessment of Educational Progress + Suchen Sie Ihr Suchwort? Bewertung; Test content; Testaufgabe; Testentwicklung; Education; assessment; Bewertungssystem
Abstract	Scoring procedures for the constructed-response (CR) items in large-scale mixed-format educational assessments often involve checks for rater agreement or rater reliability. Although these analyses are important, researchers have documented rater effects that persist despite rater training and that are not always detected in rater agreement and reliability analyses, such as severity/leniency, centrality/extremism, and biases. Left undetected, these effects pose threats to fairness. We illustrate how rater effects analyses can be incorporated into scoring procedures for large-scale mixed-format assessments. We used data from the National Assessment of Educational Progress (NAEP) to illustrate relatively simple analyses that can provide insight into patterns of rater judgment that may warrant additional attention. Our results suggested that the NAEP raters exhibited generally defensible psychometric properties, while also exhibiting some idiosyncrasies that could inform scoring procedures. Similar procedures could be used operationally to inform the interpretation and use of rater judgments in large-scale mixed-format assessments. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01