Using Vision Language Models to Detect Students’ Academic Emotion through Facial Expressions

Deliang Wang, Chao Yang, Gaowei Chen Faculty of Education, The University of Hong Kong, Hong Kong · 2025

Read Using Vision Language Models to Detect Students’ Academic Emotion through Facial Expressions by Deliang Wang, Chao Yang, Gaowei Chen Faculty of Education, The University of Hong Kong, Hong Kong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10334.txt.

Introduction

Deliang Wang, Chao Yang, and Gaowei Chen tested this 2025 idea: reading students' feelings from their faces with vision-language models. Emotions like confusion shape how students learn online. Older detectors need fresh labeled data for each new setting. Vision-language models can instead answer from a prompt alone, with no retraining.

The team ran two open models on five thousand images showing confused, distracted, happy, neutral, and tired students. Both models did moderately well overall. The Qwen model beat the Llama model, and both read happiness easily. Both failed on distracted faces, a notable gap since drifting attention matters in class. The Qwen model also read confusion fairly well, hinting at classroom uses for spotting stuck students.

Catalog record

CREATORDeliang Wang, Chao Yang, Gaowei Chen Faculty of Education, The University of Hong Kong, Hong Kong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-10334

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Using Vision Language Models to Detect Students’ Academic Emotion through Facial Expressions Deliang Wang, Chao Yang, Gaowei Chen Faculty of Education, The University of Hong Kong, Hong Kong Abstract Students’ academic emotions significantly influence their social behavior and learning performance. Traditional approaches to automatically and accurately analyze these emotions have predominantly relied on supervised machine learning algorithms. However, these models often struggle to generalize across different contexts, necessitating repeated cycles of data collection, annotation, and training. The emergence of Vision-Language Models (VLMs) offers a promising alternative, enabling generalization across visual recognition tasks through zero-shot prompting without requiring fine-tuning. This study investigates the potential of VLMs to analyze students’ academic emotions via facial expressions in an online learning environment. We employed two VLMs, Llama-3.2-11B-Vision-Instruct and Qwen2.5-VL-7B-Instruct, to analyze 5,000 images depicting confused, distracted, happy, neutral, and tired expressions using zero-shot prompting. Preliminary results indicate that both models demonstrate moderate performance in academic facial expression recognition, with Qwen2.5-VL-7B-Instruct outperforming Llama-3.2-11B-Vision-Instruct. Notably, both models excel in identifying students’ happy emotions but fail to detect distracted behavior. Additionally, Qwen2.5-VL-7B-Instruct exhibits relatively high performance in recognizing students’ confused expressions, highlighting its potential for practical applications in identifying content that causes student confusion. keywords Academic emotion, facial expression, vision language model, Llama-3.2-11B-Vision-Instruct, Qwen2.5-VL-7B-Instruct 1 Introduction Academic emotions refer to the emotions and feelings experienced by students in educational settings, such as enjoyment of learning, pride in success, and test anxiety [ 23 ] . These emotions have been found to significantly impact students’ learning outcomes [ 2 , 9 , 25 ] . For instance, a meta-analysis indicates a significantly positive correlation between positive emotions and academic achievement, and a significantly negative correlation between negative emotions and academic achievement [ 17 ] . Moreover, extreme negative academic emotions, such as depression resulting from academic stress, can lead to self-harming behaviors and, in the most severe cases, suicide [ 12 ] . Given the significance of academic emotions, researchers have explored various methods to assess these emotions in students. Traditionally, questionnaires have been employed as self-report tools for students to disclose their academic emotions [ 7 , 21 ] . However, due to fear of negative consequences, such as criticism from teachers and parents, students may sometimes conceal their true emotions and report their feelings in a more positive light. Additionally, static questionnaires are limited in their ability to capture the dynamic fluctuations of emotions during academic activities [ 16 ] . To address these limitations, researchers have begun to evaluate students’ academic emotions through observation using multimodal approaches [ 1 , 8 , 32 ] . For instance, physiological signals have been employed as indicators to reflect students’ emotional states and their variations during learning [ 15 , 27 ] . However, the high cost of biological sensors and the impracticality of requiring students to wear these sensors consistently pose significant challenges for schools. In addition to physiological signals, other modalities such as speech, text, images, and videos have been collected to analyze students’ academic emotions through voice, utterances, facial expressions, gestures, and postures [ 18 , 32 ] . Faced with large volumes of data, researchers initially adopted traditional machine learning algorithms, such as support vector machines, decision trees, and random forests, to classify students’ academic emotions [ 18 , 24 ] . Although these methods provided automated analysis, their performance required further improvement. With advancements in deep learning, deep neural networks, including convolutional neural networks (CNNs), long short-term memory networks (LSTMs), and Transformers, have been utilized to detect students’ text sentiment, facial expressions, body gestures, and postures, thereby enhancing the analysis of academic emotions [ 3 , 10 , 18 , 28 ] . Despite the outstanding performance of deep learning models, they may not generalize well across different contexts. For example, a model trained to recognize facial expressions may not be effective for detecting gestures and postures. Furthermore, models trained on Asian students may not perform adequately when applied to students from Western countries due to their differences in facial features. With the advancement of artificial intelligence (AI), the emergence of vision-language models (VLMs) presents a promising solution to the aforementioned issues. Pre-trained on large-scale datasets of image-text pairs, VLMs are multimodal models capable of performing various visual recognition tasks via zero-shot prompting without the need for fine-tuning [ 30 ] . For example, VLMs can take images and text as inputs and accordingly generate text-based answers to user questions [ 4 ] . However, the application of VLMs in detecting students’ academic emotions remains largely unexplored. Thus, this study investigates the potential of two VLMs, Llama-3.2-11B-Vision-Instruct and Qwen2.5-VL-7B-Instruct, in detecting students’ academic emotions by analyzing their facial expressions. Figure 1: Five facial expressions in the OLSFED dataset [ 3 ] . Figure 2: The prompt designed for VLMs. 2 Related Work In 1998, the American Educational Research Association held an annual meeting on “The Role of Emotions in Students’ Learning and Achievement," which marked the beginning of significant attention being paid to students’ emotions by researchers. Numerous studies have since found that students experience a variety of emotions in academic settings and have further confirmed that these academic emotions can impact students’ learning outcomes [ 2 , 9 , 20 , 25 ] . Moreover, academic emotions have also been found to influence students’ social behavior [ 12 ] . Therefore, it is crucial to accurately identify students’ emotions in academic scenarios to facilitate appropriate interventions. Traditionally, questionnaires have been the predominant method for measuring students’ academic emotions, such as the achievement emotions questionnaire [ 22 ] and the academic emotion scale [ 11 ] . However, given that questionnaires cannot capture the dynamic changes in students’ academic emotions, researchers have begun to employ technologies to dynamically track students’ academic emotions in a multimodal manner. For instance, psychophysiological measures, such as electrodermal activity and heart rate, have been used as indicators to identify and monitor students’ academic emotions in real-time during learning in computer-based learning environments [ 14 ] . In addition, students’ academic emotions can be analyzed through speech, text, images, and videos. For example, continuous speech within classroom settings has been utilized to build a speech emotion recognition system for online learning [ 6 ] . Transcripts of students’ speech can be used to trace the evolution of students’ sentiments in classroom learning [ 13 ] and computer-supported collaborative learning [ 31 ] . Images of students’ performance during learning can be used to recognize their academic emotions through facial expressions, gestures, and postures [ 5 , 18 ] . Furthermore, videos of classroom teaching can be employed to analyze students’ emotional engagement in learning [ 29 ] . To automate the analysis of academic emotions, researchers commonly use supervised machine learning techniques, including traditional algorithms and deep learning algorithms, to train models. However, these models have several limitations. For example, models trained to recognize facial expressions cannot be used to analyze body gestures. Additionally, models trained on Asian students may not perform well on Western students due to differing facial traits. In such cases, researchers may need to repeat the data collection, data annotation, and model training processes. The emergence of VLMs appears to offer a solution to these

Read the complete text