Using Differential Item Functioning to Test for Interrater Reliability in Constructed Response Items

Autor/inn/en	Walker, Cindy M.; Göçer Sahin, Sakine
Titel	Using Differential Item Functioning to Test for Interrater Reliability in Constructed Response Items
Quelle	In: Educational and Psychological Measurement, 80 (2020) 4, S.808-820 (13 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Walker, Cindy M.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0013-1644
DOI	10.1177/0013164419899731
Schlagwörter	Test Bias; Interrater Reliability; Responses; Correlation; Test Theory; Test Reliability; Statistical Analysis + Suchen Sie Ihr Suchwort? Testkritik; Interrater-Reliabilität; Korrelation; Testtheorie; Testreliabilität; Statistische Analyse
Abstract	The purpose of this study was to investigate a new way of evaluating interrater reliability that can allow one to determine if two raters differ with respect to their rating on a polytomous rating scale or constructed response item. Specifically, differential item functioning (DIF) analyses were used to assess interrater reliability and compared with traditional interrater reliability measures. Three different procedures that can be used as measures of interrater reliability were compared: (1) intraclass correlation coefficient (ICC), (2) Cohen's kappa statistic, and (3) DIF statistic obtained from Poly-SIBTEST. The results of this investigation indicated that DIF procedures appear to be a promising alternative to assess the interrater reliability of constructed response items, or other polytomous types of items, such as rating scales. Furthermore, using DIF to assess interrater reliability does not require a fully crossed design and allows one to determine if a rater is either more severe, or more lenient, in their scoring of each individual polytomous item on a test or rating scale. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01