Detecting Rater Biases in Sparse Rater-Mediated Assessment Networks

Autor/inn/en	Wind, Stefanie A.; Ge, Yuan
Titel	Detecting Rater Biases in Sparse Rater-Mediated Assessment Networks
Quelle	In: Educational and Psychological Measurement, 81 (2021) 5, S.996-1022 (27 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Wind, Stefanie A.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0013-1644
DOI	10.1177/0013164420988108
Schlagwörter	Evaluators; Scoring; Data Collection; Design; Multiple Choice Tests; Student Evaluation; Test Bias + Suchen Sie Ihr Suchwort? Bewertung; Data capture; Datensammlung; Multiple choice examinations; Multiple-choice tests, Multiple-choice examinations; Multiple-Choice-Verfahren; Schulnote; Studentische Bewertung; Testkritik
Abstract	Practical constraints in rater-mediated assessments limit the availability of complete data. Instead, most scoring procedures include one or two ratings for each performance, with overlapping performances across raters or linking sets of multiple-choice items to facilitate model estimation. These incomplete scoring designs present challenges for detecting rater biases, or differential rater functioning (DRF). The purpose of this study is to illustrate and explore the sensitivity of DRF indices in realistic sparse rating designs that have been documented in the literature that include different types and levels of connectivity among raters and students. The results indicated that it is possible to detect DRF in sparse rating designs, but the sensitivity of DRF indices varies across designs. We consider the implications of our findings for practice related to monitoring raters in performance assessments. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01