Score Resolution: An Investigation of the Reliability and Validity of Resolved Scores

Autor/inn/en	Johnson, Robert L.; Penny, Jim; Fisher, Steve; Kuhs, Therese
Titel	Score Resolution: An Investigation of the Reliability and Validity of Resolved Scores
Quelle	In: Applied Measurement in Education, 16 (2003) 4, S.299-322 (24 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0895-7347
DOI	10.1207/S15324818AME1604_3
Schlagwörter	Test Reliability; Test Validity; Scores; Interrater Reliability; Student Evaluation; Scoring; Criteria; Evaluation Methods; Measurement Techniques + Suchen Sie Ihr Suchwort? Testreliabilität; Testvalidität; Interrater-Reliabilität; Schulnote; Studentische Bewertung; Bewertung; Messtechnik
Abstract	When raters assign different scores to a performance task, a method for resolving rating differences is required to report a single score to the examinee. Recent studies indicate that decisions about examinees, such as pass/fail decisions, differ across resolution methods. Previous studies also investigated the interrater reliability of operational scores formed through different resolution methods; however, reliability might have been overestimated because adjudication was conditional on raters' initial disagreement. This study used a replication design involving autonomous teams of raters to investigate the reliability associated with three forms of resolution: averaging the original raters' scores, averaging the original raters' scores with an adjudicator's score, and matching the adjudicator's score with the closest original score. This study also examined validity coefficients for resolved scores and two types of criterion scores. Findings include (a) interrater reliability was slightly higher for the resolution method that averages the scores of the original raters and the adjudicator and (b) the lowest validity coefficients were associated most frequently with the method that matches the adjudicator's score with the closest original score. (Author).
Anmerkungen	Lawrence Erlbaum Associates, Inc. 10 Industrial Avenue, Mahwah, NJ 07430-2262. Tel: 800-926-6579; Tel: 201-258-2200; Fax: 201-236-0072; e-mail: journals@erlbaum.com; Web site: https://www.LEAonline.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2017/4/10