Practical significance of item misfit and its manifestations in constructs assessed in large-scale studies.

Autor/inn/en	Fährmann, Katharina; Köhler, Carmen; Hartig, Johannes; Heine, Jörg-Henrik
Titel	Practical significance of item misfit and its manifestations in constructs assessed in large-scale studies.
Quelle	In: Large-scale assessments in education, 10 (2022) 7, 21 S.Infoseite zur Zeitschrift PDF als Volltext kostenfreie Datei Link als defekt meldenVerfügbarkeit
Beigaben	Illustrationen
Sprache	englisch
Dokumenttyp	online; Zeitschriftenaufsatz
ISSN	2196-0739
DOI	10.1186/s40536-022-00124-w
Schlagwörter	Kompetenz; Evaluation; Korrelationsanalyse; Skalierung; Item-Response-Theory; Messung; Modell; Skalierung; Evaluation; Datenanalyse; Kompetenz; Leistungsbeurteilung; Korrelationsanalyse; Leistungsbeurteilung; Datenanalyse; Lesefertigkeit; Messung; Modell; PISA (Programme for International Student Assessment)
Abstract	When scaling psychological tests with methods of item response theory it is necessary to investigate to what extent the responses correspond to the model predictions. In addition to the statistical evaluation of item misfit, the question arises as to its practical significance. Although item removal is undesirable for several reasons, its practical consequences are rarely investigated and focus mostly on main survey data with pre-selected items. In this paper, we identify criteria to evaluate practical significance and discuss them with respect to various types of assessments and their particular purposes. We then demonstrate the practical consequences of item misfit using two data examples from the German PISA 2018 field trial study: one with cognitive data and one with non-cognitive/metacognitive data. For the former, we scale the data under the GPCM with and without the inclusion of misfitting items, and investigate how this influences the trait distribution and the allocation to reading competency levels. For non-cognitive/metacognitive data, we explore the effect of excluding misfitting items on estimated gender differences. Our results indicate minor practical consequences for person allocation and no changes in the estimated gender-difference effects. (DIPF/Orig.).
Erfasst von	DIPF \| Leibniz-Institut für Bildungsforschung und Bildungsinformation, Frankfurt am Main
Update	2023/1