AutoML Feature Engineering for Student Modeling Yields High Accuracy, but Limited Interpretability

Autor/in	Bosch, Nigel
Titel	AutoML Feature Engineering for Student Modeling Yields High Accuracy, but Limited Interpretability
Quelle	In: Journal of Educational Data Mining, 13 (2021) 2, S.55-79 (25 Seiten) PDF als Volltext kostenfreie Datei Verfügbarkeit
Zusatzinformation	ORCID (Bosch, Nigel)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	2157-2100
Schlagwörter	Accuracy; Learning Analytics; Models; National Competency Tests; Comparative Analysis; Data Analysis; Hypothesis Testing; Computer Software; Competition; Specialists; Decision Making; Evaluators; Prediction; Time Management; Mathematics Tests; Grade 4; Grade 8; National Assessment of Educational Progress + Suchen Sie Ihr Suchwort? Analogiemodell; Auswertung; Hypothesenprüfung; Hypothesentest; Wettkampf; Decision-making; Entscheidungsfindung; Vorhersage; Zeitmanagement; School year 04; 4. Schuljahr; Schuljahr 04; School year 08; 8. Schuljahr; Schuljahr 08
Abstract	Automatic machine learning (AutoML) methods automate the time-consuming, feature-engineering process so that researchers produce accurate student models more quickly and easily. In this paper, we compare two AutoML feature engineering methods in the context of the National Assessment of Educational Progress (NAEP) data mining competition. The methods we compare, Featuretools and TSFRESH (Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests), have rarely been applied in the context of student interaction log data. Thus, we address research questions regarding the accuracy of models built with AutoML features, how AutoML feature types compare to each other and to expert-engineered features, and how interpretable the features are. Additionally, we developed a novel feature selection method that addresses problems applying AutoML feature engineering in this context, where there were many heterogeneous features (over 4,000) and relatively few students. Our entry to the NAEP competition placed 3rd overall on the final held-out dataset and 1st on the public leaderboard, with a final Cohen's kappa = 0.212 and area under the receiver operating characteristic curve (AUC) = 0.665 when predicting whether students would manage their time effectively on a math assessment. We found that TSFRESH features were significantly more effective than either Featuretools features or expert-engineered features in this context; however, they were also among the most difficult features to interpret based on a survey of six experts' judgments. Finally, we discuss the tradeoffs between effort and interpretability that arise in AutoML-based student modeling. (As Provided).
Anmerkungen	International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: https://jedm.educationaldatamining.org/index.php/JEDM
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01