Semi-Automatic Coding of Open-Ended Text Responses in Large-Scale Assessments

Autor/inn/en	Andersen, Nico; Zehner, Fabian; Goldhammer, Frank
Titel	Semi-Automatic Coding of Open-Ended Text Responses in Large-Scale Assessments
Quelle	In: Journal of Computer Assisted Learning, 39 (2023) 3, S.841-854 (14 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Andersen, Nico) ORCID (Zehner, Fabian) ORCID (Goldhammer, Frank)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0266-4909
DOI	10.1111/jcal.12717
Schlagwörter	Foreign Countries; Achievement Tests; International Assessment; Secondary School Students; Educational Assessment; Coding; Simulation; Accuracy; Semantics; Program for International Student Assessment + Suchen Sie Ihr Suchwort? Ausland; Achievement test; Achievement; Testing; Test; Tests; Leistungsbeurteilung; Leistungsüberprüfung; Leistung; Testdurchführung; Testen; Sekundarschüler; Education; assessment; Bewertungssystem; Codierung; Programmierung; Simulation program; Simulationsprogramm; Semantik
Abstract	Background: In the context of large-scale educational assessments, the effort required to code open-ended text responses is considerably more expensive and time-consuming than the evaluation of multiple-choice responses because it requires trained personnel and long manual coding sessions. Aim: Our semi-supervised coding method eco (exploring coding assistant) dynamically supports human raters by automatically coding a subset of the responses. Method: We map normalized response texts into a semantic space and cluster response vectors based on their semantic similarity. Assuming that similar codes represent semantically similar responses, we propagate codes to responses in optimally homogeneous clusters. Cluster homogeneity is assessed by strategically querying informative responses and presenting them to a human rater. Following each manual coding, the method estimates the code distribution respecting a certainty interval and assumes a homogeneous distribution if certainty exceeds a predefined threshold. If a cluster is determined to certainly comprise homogeneous responses, all remaining responses are coded accordingly automatically. We evaluated the method in a simulation using different data sets. Results: With an average miscoding of about 3%, the method reduced the manual coding effort by an average of about 52%. Conclusion: Combining the advantages of automatic and manual coding produces considerable coding accuracy and reduces the required manual effort. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01