Exploring the Influence of Judge Proficiency on Standard-Setting Judgments

Autor/inn/en	Peabody, Michael R.; Wind, Stefanie A.
Titel	Exploring the Influence of Judge Proficiency on Standard-Setting Judgments
Quelle	In: Journal of Educational Measurement, 56 (2019) 1, S.101-120 (20 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0022-0655
DOI	10.1111/jedm.12202
Schlagwörter	Standard Setting; Decision Making; Performance Based Assessment; Evaluators; Cutting Scores; Expertise; Test Validity; Test Items; Item Analysis + Suchen Sie Ihr Suchwort? Standardisierung; Decision-making; Entscheidungsfindung; Leistungsermittlung; Expert appraisal; Testvalidität; Test content; Testaufgabe; Itemanalyse
Abstract	Setting performance standards is a judgmental process involving human opinions and values as well as technical and empirical considerations. Although all cut score decisions are by nature somewhat arbitrary, they should not be capricious. Judges selected for standard-setting panels should have the proper qualifications to make the judgments asked of them; however, even qualified judges vary in expertise and in some cases, such as highly specialized areas or when members of the public are involved, it may be difficult to ensure that each member of a standard-setting panel has the requisite expertise to make qualified judgments. Given the subjective nature of these types of judgments, and that a large part of the validity argument for an exam lies in the robustness of its passing standard, an examination of the influence of judge proficiency on the judgments is warranted. This study explores the use of the many-facet Rasch model as a method for adjusting modified Angoff standard-setting ratings based on judges' proficiency levels. The results suggest differences in the severity and quality of standard-setting judgments across levels of judge proficiency, such that judges who answered easy items incorrectly tended to perceive them as easier, but those who answered correctly tended to provide ratings within normal stochastic limits. (As Provided).
Anmerkungen	Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: cs-journals@wiley.com; Web site: http://www.wiley.com/WileyCDA
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01