The NAEP EDM Competition. Theory-Driven Psychometrics and Machine Learning for Predictions Based on Log Data.

Autor/inn/en	Zehner, Fabian; Harrison, Scott; Eichmann, Beate; Deribo, Tobias; Bengs, Daniel; Andersen, Nico; Hahnel, Carolin
Titel	The NAEP EDM Competition. Theory-Driven Psychometrics and Machine Learning for Predictions Based on Log Data.
Quelle	Aus: Rafferty, Anna N. (Hrsg.); Whitehill, Jacob (Hrsg.); Cavalli-Sforza, Violetta (Hrsg.); Romero, Cristobal (Hrsg.): Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020). Worcester, MA: International Educational Data Mining Society (2020) S. 302-312 PDF als Volltext kostenfreie Datei Link als defekt meldenVerfügbarkeit
Beigaben	Literaturangaben
Sprache	englisch
Dokumenttyp	online; gedruckt; Sammelwerksbeitrag
ISBN	978-1-7336736-1-7
Schlagwörter	Reliabilität; Validität; Psychometrie; Testauswertung; Testdurchführung; Testergebnis; Künstliche Intelligenz; Lernen; Lernforschung; Datenanalyse; Datenerfassung; Datengewinnung; Prozessdatenverarbeitung; Technologiebasiertes Testen; Effizienz; Klassifikationssystem; Modell; Modellierung; Statistische Methode; Verfahren; Datenaufbereitung
Abstract	The 2nd Annual WPI-UMASS-UPENN EDM Data Mining Challenge required contestants to predict efficient test-taking based on log data. In this paper, [the authors] describe [their] theory-driven and psychometric modeling approach. For feature engineering, [they] employed the Log-Normal Response Time Model for estimating latent person speed, and the Generalized Partial Credit Model for estimating latent person ability. Additionally, [the authors] adopted ann-gram feature approach for event sequences. For training a multi-label classifier, [the authors] distinguished inefficient test takers who were going too fast and those who were going too slow, instead of using the provided binary target label. [Their] best-performing ensemble classifier comprised three sets of low-dimensional classifiers, dominated by test-taker speed. While [their] classifier reached moderate performance, relative to competition leader board, [their] approach makes two important contributions. First, [the authors] show how explainable classifiers could provide meaningful predictions if results can be contextualized to test administrators who wish to intervene or take action. Second, [their] re-engineering of test scores enabled [them] to incorporate person ability into the estimation. However, ability was hardly predictive of efficient behavior, leading to the conclusion that the target label's validity needs to be questioned. The paper concludes with tools that are helpful for substantively meaningful log data mining. (Orig.).
Erfasst von	DIPF \| Leibniz-Institut für Bildungsforschung und Bildungsinformation, Frankfurt am Main
Update	2021/2