Automated Assessment of Student Hand Drawings in Free-Response Items on the Particulate Nature of Matter

Autor/inn/en	Lee, Jaeyong; Lee, Gyeong-Geon; Hong, Hun-Gi
Titel	Automated Assessment of Student Hand Drawings in Free-Response Items on the Particulate Nature of Matter
Quelle	In: Journal of Science Education and Technology, 32 (2023) 4, S.549-566 (18 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Hong, Hun-Gi)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1059-0145
DOI	10.1007/s10956-023-10042-3
Schlagwörter	Freehand Drawing; Evaluation Methods; Elementary School Students; Secondary School Students; Foreign Countries; Semantics; Scores; Science Tests; Responses; Visual Aids; Artificial Intelligence; Scientific Concepts; Classification; Student Evaluation; Science Instruction; Dictionaries; Recall (Psychology); Instructional Program Divisions; Accuracy; South Korea + Suchen Sie Ihr Suchwort? Drawing; Zeichnen; Sekundarschüler; Ausland; Semantik; Anschauungsmaterial; Künstliche Intelligenz; Classification system; Klassifikation; Klassifikationssystem; Schulnote; Studentische Bewertung; Teaching of science; Science education; Natural sciences Lessons; Naturwissenschaftlicher Unterricht; Dictionary; Wörterbuch; Abberufung; Korea; Republik
Abstract	Here, we describe the development and validation of an automatic assessment system that examines students' hand-drawn visual representations in free-response items. The data were collected from 1,028 students in the second through 11th grades in South Korea using two items from the Test About Particles in a Gas questionnaire (Novick & Nussbaum, 1981). Students' free responses, which include hand drawings and writing, were coded for two dimensions--structural (particulate/continuous/other) and distributional (expanded/concentrated/other). Machine learning (ML) models were trained to assess the responses on the particulate nature of matter. For classifying hand drawings, a pre-trained Inception-v3 model followed by a support vector machine was trained and its performance was evaluated. The assessment model yielded high machine-human agreement (MHA) (kappa = 0.732-0.926, accuracy = 0.820-0.942, precision = 0.817-0.941, recall = 0.820-0.942, F1 = 0.818-0.941, and area under the curve [AUC] = 0.906-0.990). Students' written responses were tokenized, and a dictionary of scientific semantic scores was prepared. The final model for the overall assessment of both drawing and writing yielded high MHA (kappa = 0.800-0.881, accuracy = 0.859-0.956, precision = 0.865-0.957, recall = 0.859-0.956, F1 = 0.859-0.956, and AUC = 0.944-0.995), which varied by the final classifiers of the models. There were some variances in the performance of the assessment model according to the school level. This study suggests that artificial intelligence can be used to automate assessments of students' representations of scientific concepts in free-response items, particularly those drawn in a pencil-and-paper format. (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01