Complementary Strengths? Evaluation of a Hybrid Human-Machine Scoring Approach for a Test of Oral Academic English

Autor/inn/en	Davis, Larry; Papageorgiou, Spiros
Titel	Complementary Strengths? Evaluation of a Hybrid Human-Machine Scoring Approach for a Test of Oral Academic English
Quelle	In: Assessment in Education: Principles, Policy & Practice, 28 (2021) 4, S.437-455 (19 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Davis, Larry) ORCID (Papageorgiou, Spiros)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0969-594X
DOI	10.1080/0969594X.2021.1979466
Schlagwörter	Scoring; English for Academic Purposes; Oral English; Speech Tests; Man Machine Systems; English (Second Language); Language Tests; Second Language Learning; Automation; Test Reliability; Test of English as a Foreign Language + Suchen Sie Ihr Suchwort? Bewertung; Sprechtest; Mensch-Maschine-System; English as second language; English; Second Language; Englisch als Zweitsprache; Language test; Sprachtest; Zweitsprachenerwerb; Testreliabilität
Abstract	Human raters and machine scoring systems potentially have complementary strengths in evaluating language ability; specifically, it has been suggested that automated systems might be used to make consistent measurements of specific linguistic phenomena, whilst humans evaluate more global aspects of performance. We report on an empirical study that explored the possibility of combining human and machine scores using responses from the speaking section of the TOEFL iBT® test. Human raters awarded scores for three sub-constructs: delivery, language use and topic development. The SpeechRater[superscript SM] automated scoring system produced scores for delivery and language use. Composite scores computed from three different combinations of human and automated analytic scores were equally or more reliable than human holistic scores, probably due to the inclusion of multiple observations in composite scores. However, composite scores calculated solely from human analytic scores showed the highest reliability and reliability steadily decreased as more machine scores replaced human scores. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01