Examining the Impacts of Ignoring Rater Effects in Mixed-Format Tests

Autor/inn/en	Guo, Wenjing; Wind, Stefanie A.
Titel	Examining the Impacts of Ignoring Rater Effects in Mixed-Format Tests
Quelle	In: Journal of Educational Measurement, 58 (2021) 3, S.364-387 (24 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Guo, Wenjing) ORCID (Wind, Stefanie A.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0022-0655
DOI	10.1111/jedm.12292
Schlagwörter	Test Format; Multiple Choice Tests; Scoring; Test Items; Responses; National Competency Tests; Academic Achievement; Computation; National Assessment of Educational Progress + Suchen Sie Ihr Suchwort? Testentwicklung; Multiple choice examinations; Multiple-choice tests, Multiple-choice examinations; Multiple-Choice-Verfahren; Bewertung; Test content; Testaufgabe; Schulleistung
Abstract	The use of mixed-format tests made up of multiple-choice (MC) items and constructed response (CR) items is popular in large-scale testing programs, including the National Assessment of Educational Progress (NAEP) and many district- and state-level assessments in the United States. Rater effects, or raters' scoring tendencies that result in performances receiving different scores than are warranted given their quality, are concerns for the interpretation of scores on CR items. However, there are few published studies in which researchers have systematically considered the impact of ignoring rater effects when they are present on estimates of student ability using large-scale mixed-format assessments. Using results from an analysis of NAEP data, we systematically explored the impacts of rater effects on student achievement estimates. Our results suggest that in conditions that reflect many large-scale mixed-format assessments, directly modeling rater effects yields more accurate student achievement estimates than estimation procedures that do not incorporate raters. We consider the implications of our findings for research and practice. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01