From byproduct to design factor. On validating the interpretation of process indicators based on log data.

Autor/inn/en	Goldhammer, Frank; Hahnel, Carolin; Kröhne, Ulf; Zehner, Fabian
Titel	From byproduct to design factor. On validating the interpretation of process indicators based on log data.
Quelle	In: Large-scale assessments in education, 9 (2021) 20, S. 9-20Infoseite zur Zeitschrift PDF als Volltext kostenfreie Datei (1); PDF als Volltext kostenfreie Datei (2); PDF als Volltext kostenfreie Datei (3) Link als defekt meldenVerfügbarkeit
Beigaben	Illustrationen
Sprache	englisch
Dokumenttyp	online; Zeitschriftenaufsatz
ISSN	2196-0739
DOI	10.1186/s40536-021-00113-5 10.25656/01:25005
URN	urn:nbn:de:0111-pedocs-250050
Schlagwörter	Testkonstruktion; Testtheorie; Typologie; Interpretation; Datenanalyse; Testkonstruktion; Testtheorie; Daten; Leistungsbeurteilung; Indikator; Leistungsmessung; Logdatei; Leistungsbeurteilung; Indikator; Datenanalyse; Messverfahren; Daten; Interpretation; Leistungsmessung; Typologie; Logdatei; PIAAC (Programme for the International Assessment of Adult Competencies); PISA (Programme for International Student Assessment)
Abstract	International large-scale assessments such as PISA or PIAAC have started to provide public or scientific use files for log data; that is, events, event-related attributes and timestamps of test-takers' interactions with the assessment system. Log data and the process indicators derived from it can be used for many purposes. However, the intended uses and interpretations of process indicators require validation, which here means a theoretical and/or empirical justification that inferences about (latent) attributes of the test-taker's work process are valid. This article reviews and synthesizes measurement concepts from various areas, including the standard assessment paradigm, the continuous assessment approach, the evidence-centered design (ECD) framework, and test validation. Based on this synthesis, we address the questions of how to ensure the valid interpretation of process indicators by means of an evidence-centered design of the task situation, and how to empirically challenge the intended interpretation of process indicators by developing and implementing correlational and/or experimental validation strategies. For this purpose, we explicate the process of reasoning from log data to low-level features and process indicators as the outcome of evidence identification. In this process, contextualizing information from log data is essential in order to reduce interpretative ambiguities regarding the derived process indicators. Finally, we show that empirical validation strategies can be adapted from classical approaches investigating the nomothetic span and construct representation. Two worked examples illustrate possible validation strategies for the design phase of measurements and their empirical evaluation. (DIPF/Orig.).
Erfasst von	DIPF \| Leibniz-Institut für Bildungsforschung und Bildungsinformation, Frankfurt am Main
Update	2022/2