Literaturnachweis - Detailanzeige
| Autor/inn/en | Doewes, Afrizal; Kurdhi, Nughthoh Arfawi; Saxena, Akrati |
|---|---|
| Titel | Evaluating Quadratic Weighted Kappa as the Standard Performance Metric for Automated Essay Scoring [Konferenzbericht] Paper presented at the International Conference on Educational Data Mining (EDM) (16th, Bengaluru, India, Jul 11-14, 2023). |
| Quelle | (2023), (11 Seiten)
PDF als Volltext kostenfreie Datei |
| Sprache | englisch |
| Dokumenttyp | gedruckt; online; Monographie |
| Schlagwörter | Essays; Writing Evaluation; Evaluators; Accuracy; Models; Computer Assisted Testing; Rating Scales; High Stakes Tests; Interrater Reliability; Computer Software; Artificial Intelligence |
| Abstract | Automated Essay Scoring (AES) tools aim to improve the efficiency and consistency of essay scoring by using machine learning algorithms. In the existing research work on this topic, most researchers agree that human-automated score agreement remains the benchmark for assessing the accuracy of machine-generated scores. To measure the performance of AES models, the Quadratic Weighted Kappa (QWK) is commonly used as the evaluation metric. However, we have identified several limitations of using QWK as the sole metric for evaluating AES model performance. These limitations include its sensitivity to the rating scale, the potential for the so-called "kappa paradox" to occur, the impact of prevalence, the impact of the position of agreements in the diagonal agreement matrix, and its limitation in handling a large number of raters. Our findings suggest that relying solely on QWK as the evaluation metric for AES performance may not be sufficient. We further discuss insights into additional metrics to comprehensively evaluate the performance and accuracy of AES models. [For the complete proceedings, see ED630829.] (As Provided). |
| Anmerkungen | International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/ |
| Erfasst von | ERIC (Education Resources Information Center), Washington, DC |
| Update | 2024/1/01 |
Literaturbeschaffung und Bestandsnachweise in Bibliotheken prüfen
Permalink als QR-Code
Inhalt auf sozialen Plattformen teilen (nur vorhanden, wenn Javascript eingeschaltet ist)
Teile diese Seite:
