To score or not to score. Factors influencing performance and feasibility of automatic content scoring of text responses.

Autor/inn/en	Zesch, Torsten; Horbach, Andrea
Titel	To score or not to score. Factors influencing performance and feasibility of automatic content scoring of text responses.
Quelle	In: Educational measurement, 42 (2023) 1, S. 44-58Infoseite zur Zeitschrift PDF als Volltext kostenfreie Datei Link als defekt meldenVerfügbarkeit
Beigaben	Illustrationen
Sprache	englisch
Dokumenttyp	online; Zeitschriftenaufsatz
ISSN	1745-3992
DOI	10.1111/emip.12544
Schlagwörter	Bewertung; Text; Leistung; Inhalt; Verfahren; Antwort; Text; Automatisierung; Antwort; Bewertung; Einflussfaktor; Inhalt; Leistung; Verfahren; Tool
Abstract	In this article, we systematize the factors influencing performance and feasibility of automatic content scoring methods for short text responses. We argue that performance (i.e., how well an automatic system agrees with human judgments) mainly depends on the linguistic variance seen in the responses and that this variance is indirectly influenced by other factors such as target population or input modality. Extending previous work, we distinguish conceptual, realization, and nonconformity variance, which are differentially impacted by the various factors. While conceptual variance relates to different concepts embedded in the text responses, realization variance refers to their diverse manifestation through natural language. Nonconformity variance is added by aberrant response behavior. Furthermore, besides its performance, the feasibility of using an automatic scoring system depends on external factors, such as ethical or computational constraints, which influence whether a system with a given performance is accepted by stakeholders. Our work provides (i) a framework for assessment practitioners to decide a priori whether automatic content scoring can be successfully applied in a given setup as well as (ii) new empirical findings and the integration of empirical findings from the literature on factors that influence automatic systems' performance. (DIPF/Orig.).
Erfasst von	DIPF \| Leibniz-Institut für Bildungsforschung und Bildungsinformation, Frankfurt am Main
Update	2023/1