Automatic Coding of Short Text Responses via Clustering in Educational Assessment

Autor/inn/en	Zehner, Fabian; Sälzer, Christine; Goldhammer, Frank
Titel	Automatic Coding of Short Text Responses via Clustering in Educational Assessment
Quelle	In: Educational and Psychological Measurement, 76 (2016) 2, S.280-303 (24 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0013-1644
DOI	10.1177/0013164415590022
Schlagwörter	Educational Assessment; Coding; Automation; Responses; Cluster Grouping; Natural Language Processing; Accuracy; International Assessment; Achievement Tests; Secondary School Students; Foreign Countries; Test Items; Sample Size; Computer Software; Computer Assisted Testing; Germany; Program for International Student Assessment + Suchen Sie Ihr Suchwort? Education; assessment; Bewertungssystem; Codierung; Programmierung; Eingruppierung; Natürliche Sprache; Achievement test; Achievement; Testing; Test; Tests; Leistungsbeurteilung; Leistungsüberprüfung; Leistung; Testdurchführung; Testen; Sekundarschüler; Ausland; Test content; Testaufgabe; Deutschland
Abstract	Automatic coding of short text responses opens new doors in assessment. We implemented and integrated baseline methods of natural language processing and statistical modelling by means of software components that are available under open licenses. The accuracy of automatic text coding is demonstrated by using data collected in the "Programme for International Student Assessment" (PISA) 2012 in Germany. Free text responses of 10 items with n = 41,990 responses in total were analyzed. We further examined the effect of different methods, parameter values, and sample sizes on performance of the implemented system. The system reached fair to good up to excellent agreement with human codings (0.458 = ? = 0.959). Especially items that are solved by naming specific semantic concepts appeared properly coded. The system performed equally well with n = 1 , 661 and somewhat poorer but still acceptable down to n = 249 . Based on our findings, we discuss potential innovations for assessment that are enabled by automatic coding of short text responses. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01