Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches across Constructed Response Assessments Aligned to a Science Learning Progression

Autor/inn/en	Jescovitch, Lauren N.; Scott, Emily E.; Cerchiara, Jack A.; Merrill, John; Urban-Lurain, Mark; Doherty, Jennifer H.; Haudek, Kevin C.
Titel	Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches across Constructed Response Assessments Aligned to a Science Learning Progression
Quelle	In: Journal of Science Education and Technology, 30 (2021) 2, S.150-167 (18 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Jescovitch, Lauren N.) ORCID (Scott, Emily E.) ORCID (Cerchiara, Jack A.) ORCID (Urban-Lurain, Mark) ORCID (Doherty, Jennifer H.) ORCID (Haudek, Kevin C.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1059-0145
DOI	10.1007/s10956-020-09858-0
Schlagwörter	Science Instruction; Coding; Artificial Intelligence; Man Machine Systems; Holistic Approach; Logical Thinking; Physiology; Learning Processes; Responses; Classification; Undergraduate Students; Scoring Rubrics; Models + Suchen Sie Ihr Suchwort? Teaching of science; Science education; Natural sciences Lessons; Naturwissenschaftlicher Unterricht; Codierung; Programmierung; Künstliche Intelligenz; Mensch-Maschine-System; Holistischer Ansatz; Physiologie; Learning process; Lernprozess; Classification system; Klassifikation; Klassifikationssystem; Scoring formulas; Auswertungsbogen; Analogiemodell
Abstract	We systematically compared two coding approaches to generate training datasets for machine learning (ML): (1) a holistic approach based on learning progression levels; and (2) a dichotomous, analytic approach of multiple concepts in student reasoning, deconstructed from holistic rubrics. We evaluated four constructed response assessment items for undergraduate physiology, each targeting five levels of a developing flux learning progression in an ion context. Human-coded datasets were used to train two ML models: (1) an 8-classification algorithm ensemble implemented in the Constructed Response Classifier (CRC); and (2) a single classification algorithm implemented in LightSide Researcher's Workbench. Human coding agreement on approximately 700 student responses per item was high for both approaches with Cohen's kappas ranging from 0.75 to 0.87 on holistic scoring and from 0.78 to 0.89 on analytic composite scoring. ML model performance varied across items and rubric type. For two items, training sets from both coding approaches produced similarly accurate ML models, with differences in Cohen's kappa between machine and human scores of 0.002 and 0.041. For the other items, ML models trained with analytic coded responses and used for a composite score, achieved better performance as compared to using holistic scores for training, with increases in Cohen's kappa of 0.043 and 0.117. These items used a more complex scenario involving movement of two ions. It may be that analytic coding is beneficial to unpacking this additional complexity. (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01