Multichannel assessment of cognitive load and emotions with pedagogical avatars in higher education

Abstract

Integrating multimodal devices (eye tracking, electroencephalography [EEG], and galvanic skin response [GSR]) enables the objective assessment of cognitive and emotional processes during controlled tasks. In higher education multimedia contexts, such measures can inform the adaptive design of educational interfaces. This study examined the processing of avatar images expressing positive (surprise, joy) and negative (sadness, fear) emotions across different user groups (final-year university students, novice university lecturers, and experienced university lecturers), while also considering gender effects. Forty-eight participants completed a visualisation task while physiological indicators derived from EEG (engagement, memorisation, valence, workload), GSR (impact), and eye tracking (number of visits and dwell time) were recorded for relevant and non-relevant areas of interest. Significant differences emerged for surprise, with students showing more visits to non-relevant areas than lecturers members. For joy, final-year university students exhibited longer dwell times on relevant areas than experienced lecturers. No differences by professional group were found for sadness or fear. Regarding gender, women showed higher valence and workload in the joy condition, as well as higher memorisation and longer dwell times in the sadness condition. Based on these findings, adaptive adjustments to avatar design are proposed to better align with users’ cognitive and emotional profiles.