Associating Facial Expressions and Upper-Body Gestures with Learning Tasks for Enhancing Intelligent Tutoring Systems

Autor/inn/en	Behera, Ardhendu; Matthew, Peter; Keidel, Alexander; Vangorp, Peter; Fang, Hui; Canning, Susan
Titel	Associating Facial Expressions and Upper-Body Gestures with Learning Tasks for Enhancing Intelligent Tutoring Systems
Quelle	In: International Journal of Artificial Intelligence in Education, 30 (2020) 2, S.236-270 (35 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Behera, Ardhendu)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1560-4292
DOI	10.1007/s40593-020-00195-2
Schlagwörter	Nonverbal Communication; Intelligent Tutoring Systems; Eye Movements; Learning Processes; Video Technology; Undergraduate Students; Computer Science Education; Problem Solving; Difficulty Level; Behavior Patterns; Artificial Intelligence + Suchen Sie Ihr Suchwort? Non-verbal communication; Nonverbale Kommunikation; Intelligentes Tutorsystem; Augenbewegung; Learning process; Lernprozess; Computer science lessons; Informatikunterricht; Problemlösen; Schwierigkeitsgrad; Künstliche Intelligenz
Abstract	Learning involves a substantial amount of cognitive, social and emotional states. Therefore, recognizing and understanding these states in the context of learning is key in designing informed interventions and addressing the needs of the individual student to provide personalized education. In this paper, we explore the automatic detection of learner's nonverbal behaviors involving hand-over-face gestures, head and eye movements and emotions via facial expressions during learning. The proposed computer vision-based behavior monitoring method uses a low-cost webcam and can easily be integrated with modern tutoring technologies. We investigate these behaviors in-depth over time in a classroom session of 40 minutes involving reading and problem-solving exercises. The exercises in the sessions are divided into three categories: an easy, medium and difficult topic within the context of undergraduate computer science. We found that there is a significant increase in head and eye movements as time progresses, as well as with the increase of difficulty level. We demonstrated that there is a considerable occurrence of "hand-over-face gestures" (on average 21.35%) during the 40 minutes session and is unexplored in the education domain. We propose a novel deep learning approach for automatic detection of "hand-over-face" gestures in images with a classification accuracy of 86.87%. There is a prominent increase in "hand-over-face" gestures when the difficulty level of the given exercise increases. The "hand-over-face" gestures occur more frequently during problem-solving (easy 23.79%, medium 19.84% and difficult 30.46%) exercises in comparison to reading (easy 16.20%, medium 20.06% and difficult 20.18%). (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01