Automated Essay Scoring at Scale: A Case Study in Switzerland and Germany.

Autor/inn/en	Rupp, André A.; Casabianca, Jodi M.; Krüger, Maleika; Keller, Stefan; Köller, Olaf
Titel	Automated Essay Scoring at Scale: A Case Study in Switzerland and Germany.
Quelle	Princeton, NJ: Educational Testing Service (2019), 21 S. PDF als Volltext Link als defekt meldenVerfügbarkeit
Reihe	ETS Research Report Series. RR-19-12; TOEFL Research Report. RR-86
Beigaben	Literaturangaben
Sprache	englisch
Dokumenttyp	online; gedruckt; Monographie
ISSN	2330-8516
DOI	10.1002/ets2.12249
Schlagwörter	Längsschnittuntersuchung; Reliabilität; Vergleich; Fähigkeit; Testverfahren; Bewertung; Deutschland; Essay; Längsschnittuntersuchung; Schüler; Schweiz; Lehrerbeurteilung; Schreiben; Aufsatz; Computerunterstütztes Verfahren; Fähigkeit; Textproduktion; Vergleich; Reliabilität; Automatische Messung; Sekundarbereich; Lehrerbeurteilung; Schüler; Methodik; Aufsatz; Essay; Textproduktion; Schreiben; Automatische Messung; Computerunterstütztes Verfahren; Bewertung; Deutschland; Schweiz
Abstract	In this research report, we describe the design and empirical findings for a large-scale study of essay writing ability with approximately 2,500 high school students in Germany and Switzerland on the basis of 2 tasks with 2 associated prompts, each from a standardized writing assessment whose scoring involved both human and automated components. For the human scoring aspect, we describe the methodology for training and monitoring human raters as well as for collecting their ratings within a customized platform. For the automated scoring aspect, we describe the methodology for training, evaluating, and selecting appropriate automated scoring models as well as correlational patterns of resulting task scores with scores from secondary measures. Analyses show that the human ratings were highly reliable and that effective prompt-specific automated scoring models could be built with state-of-the-art features and machine learning methods, which resulted in correlational patterns with secondary measures that were in line with general expectations. In closing, we discuss the methodological implications for conducting this kind of work at scale in the future. (Orig.).
Erfasst von	DIPF \| Leibniz-Institut für Bildungsforschung und Bildungsinformation, Frankfurt am Main
Update	2021/1