Validating Rubric Scoring Processes: An Application of an Item Response Tree Model

Autor/inn/en	Myers, Aaron J.; Ames, Allison J.; Leventhal, Brian C.; Holzman, Madison A.
Titel	Validating Rubric Scoring Processes: An Application of an Item Response Tree Model
Quelle	In: Applied Measurement in Education, 33 (2020) 4, S.293-308 (16 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Myers, Aaron J.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0895-7347
DOI	10.1080/08957347.2020.1789143
Schlagwörter	Scoring Rubrics; Validity; Item Response Theory; Interrater Reliability; Student Evaluation; Evaluation Methods; Writing Evaluation + Suchen Sie Ihr Suchwort? Scoring formulas; Auswertungsbogen; Gültigkeit; Item-Response-Theorie; Interrater-Reliabilität; Schulnote; Studentische Bewertung
Abstract	When rating performance assessments, raters may ascribe different scores for the same performance when rubric application does not align with the intended application of the scoring criteria. Given performance assessment score interpretation assumes raters apply rubrics as rubric developers intended, misalignment between raters' scoring processes and the intended scoring processes may lead to invalid inferences from these scores. In an effort to standardize raters' scoring processes, an alternative scoring method was used. With this method, rubric developers' intended scoring processes are made explicit by requiring raters to respond to a series of selected-response statements resembling a decision tree. To determine if raters scored essays as intended using a traditional rubric and the alternative scoring method, an IRT model with a tree-like structure (IRTree) was specified to depict the intended scoring processes and fit to data from each scoring method. Results suggest raters using the alternative method may be better able to rate as intended and thus the alternative method may be a viable alternative to traditional rubric scoring. Implications of the IRTree model are discussed. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01