Evaluating the Consistency of Angoff-Based Cut Scores Using Subsets of Items within a Generalizability Theory Framework

Autor/inn/en	Kannan, Priya; Sgammato, Adrienne; Tannenbaum, Richard J.; Katz, Irvin R.
Titel	Evaluating the Consistency of Angoff-Based Cut Scores Using Subsets of Items within a Generalizability Theory Framework
Quelle	In: Applied Measurement in Education, 28 (2015) 3, S.169-186 (18 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0895-7347
DOI	10.1080/08957347.2015.1042156
Schlagwörter	Reliability; Standard Setting (Scoring); Cutting Scores; Test Items; Generalizability Theory; Probability; Error of Measurement; Test Length; Sampling; Licensing Examinations (Professions) + Suchen Sie Ihr Suchwort? Reliabilität; Bewertungsskala; Test content; Testaufgabe; Wahrscheinlichkeitsrechnung; Wahrscheinlichkeitstheorie; Messfehler
Abstract	The Angoff method requires experts to view every item on the test and make a probability judgment. This can be time consuming when there are large numbers of items on the test. In this study, a G-theory framework was used to determine if a subset of items can be used to make generalizable cut-score recommendations. Angoff ratings (i.e., probability judgments) from previously conducted standard setting studies were used first in a re-sampling study, followed by D-studies. For the re-sampling study, proportionally stratified subsets of items were extracted under various sampling and test-length conditions. The mean cut score, variance components, expected standard error (SE) around the mean cut score, and root-mean-squared deviation (RMSD) across 1,000 replications were estimated at each study condition. The SE and the RMSD decreased as the number of items increased, but this reduction tapered off after approximately 45 items. Subsequently, D-studies were performed on the same datasets. The expected SE was computed at various test lengths. Results from both studies are consistent with previous research indicating that between 40-50 items are sufficient to make generalizable cut score recommendations. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01