More Efficient Processes for Creating Automated Essay Scoring Frameworks: A Demonstration of Two Algorithms

Autor/inn/en	Shin, Jinnie; Gierl, Mark J.
Titel	More Efficient Processes for Creating Automated Essay Scoring Frameworks: A Demonstration of Two Algorithms
Quelle	In: Language Testing, 38 (2021) 2, S.247-272 (26 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Shin, Jinnie)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0265-5322
DOI	10.1177/0265532220937830
Schlagwörter	Scoring; Essays; Writing Evaluation; Computer Software; Scores; Evaluators; Models; High Stakes Tests; Networks; Comparative Analysis; Artificial Intelligence; Reliability + Suchen Sie Ihr Suchwort? Bewertung; Essay; Aufsatzunterricht; Analogiemodell; Künstliche Intelligenz; Reliabilität
Abstract	Automated essay scoring (AES) has emerged as a secondary or as a sole marker for many high-stakes educational assessments, in native and non-native testing, owing to remarkable advances in feature engineering using natural language processing, machine learning, and deep-neural algorithms. The purpose of this study is to compare the effectiveness and the performance of two AES frameworks, each based on machine learning with deep language features, or complex language features, and deep neural algorithms. More specifically, support vector machines (SVMs) in conjunction with Coh-Metrix features were used for a traditional AES model development, and the convolutional neural networks (CNNs) approach was used for more contemporary deep-neural model development. Then, the strengths and weaknesses of the traditional and contemporary models under different circumstances (e.g., types of the rubric, length of the essay, and the essay type) were tested. The results were evaluated using the quadratic weighted kappa (QWK) score and compared with the agreement between the human raters. The results indicated that the CNNs model performs better, meaning that it produced more comparable results to the human raters than the Coh-Metrix + SVMs model. Moreover, the CNNs model also achieved state-of-the-art performance in most of the essay sets with a high average QWK score. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01