A Sequential Approach to Detecting Differential Rater Functioning in Sparse Rater-Mediated Assessment Networks

Autor/in	Wind, Stefanie A.
Titel	A Sequential Approach to Detecting Differential Rater Functioning in Sparse Rater-Mediated Assessment Networks
Quelle	In: Language Testing, 40 (2023) 2, S.209-226 (18 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Wind, Stefanie A.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0265-5322
DOI	10.1177/02655322221092388
Schlagwörter	Evaluators; Decision Making; Student Characteristics; Performance Based Assessment; Academic Achievement; Item Analysis; Accuracy; Writing Evaluation; Simulation; Sequential Approach; Language Tests; English; Language Arts; High School Students; Multiple Choice Tests; Writing Tests; Case Studies; Bias + Suchen Sie Ihr Suchwort? Decision-making; Entscheidungsfindung; Leistungsermittlung; Schulleistung; Itemanalyse; Simulation program; Simulationsprogramm; Schrittfolge; Language test; Sprachtest; English language; Englisch; Sprachkultur; High school; High schools; Student; Students; Oberschule; Schüler; Schülerin; Studentin; Multiple choice examinations; Multiple-choice tests, Multiple-choice examinations; Multiple-Choice-Verfahren; Writing test; Schreibtest; Case study; Fallstudie; Case Study
Abstract	Researchers frequently evaluate rater judgments in performance assessments for evidence of differential rater functioning (DRF), which occurs when rater severity is systematically related to construct-irrelevant student characteristics after controlling for student achievement levels. However, researchers have observed that methods for detecting DRF may be limited in sparse rating designs, where it is not possible for every rater to score every student. In these designs, there is limited information with which to detect DRF. Sparse designs can also exacerbate the impact of artificial DRF, which occurs when raters are inaccurately flagged for DRF due to statistical artifacts. In this study, a sequential method is adapted from previous research on differential item functioning (DIF) that allows researchers to detect DRF more accurately and distinguish between true and artificial DRF. Analyses of data from a rater-mediated writing assessment and a simulation study demonstrate that the sequential approach results in different conclusions about which raters exhibit DRF. Moreover, the simulation study results suggest that the sequential procedure results in improved accuracy in DRF detection across a variety of rating design conditions. Practical implications for language testing research are discussed. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01