Conversation-Based Assessments in Education: Design, Implementation, and Cognitive Walkthroughs for Usability Testing

Autor/inn/en	Yildirim-Erbasli, Seyma N.; Bulut, Okan; Demmans Epp, Carrie; Cui, Ying
Titel	Conversation-Based Assessments in Education: Design, Implementation, and Cognitive Walkthroughs for Usability Testing
Quelle	In: Journal of Educational Technology Systems, 52 (2023) 1, S.27-51 (25 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Yildirim-Erbasli, Seyma N.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0047-2395
DOI	10.1177/00472395231178943
Schlagwörter	Artificial Intelligence; Intelligent Tutoring Systems; Computer Mediated Communication; Formative Evaluation; Usability; Responses; Computer Assisted Testing; Accuracy; Computer Interfaces; Man Machine Systems; Design; Undergraduate Students + Suchen Sie Ihr Suchwort? Künstliche Intelligenz; Intelligentes Tutorsystem; Computerkonferenz; Mensch-Maschine-System
Abstract	Conversational agents have been widely used in education to support student learning. There have been recent attempts to design and use conversational agents to conduct assessments (i.e., conversation-based assessments: CBA). In this study, we developed CBA with constructed and selected-response tests using Rasa--an artificial intelligence-based tool. CBA was deployed via Google Chat to support formative assessment. We evaluated (1) its performance in answering students' responses and (2) its usability with cognitive walkthroughs conducted by external evaluators. CBA with constructed-response tests consistently matched student responses to the appropriate conversation paths in most cases. In comparison, CBA with selected-response tests demonstrated perfect accuracy between system design and implementation. A cognitive walkthrough of CBA showed its usability as well as several potential issues that could be improved. Participating students did not experience these issues, however, we reported them to help researchers, designers, and practitioners improve the assessment experience for students using CBA. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01