Exploring the Application of Visual Question Answering (VQA) for Classroom Activity Monitoring
Sinh Trong Vu, Hieu Trung Pham, Dung Manh Nguyen, Hieu Minh Hoang, Nhu Hoang Le, Thu Ha Pham, Tai Tan Mai · 2025
Read Exploring the Application of Visual Question Answering (VQA) for Classroom Activity Monitoring by Sinh Trong Vu, Hieu Trung Pham, Dung Manh Nguyen, Hieu Minh Hoang, Nhu Hoang Le, Thu Ha Pham, Tai Tan Mai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22369.txt.
Introduction
Sinh Trong Vu and co-authors explore, in a 2025 paper, whether vision-language models can watch classroom videos and answer questions about student behavior. Watching classes by hand is slow, yet behavior data helps teachers lift engagement. Models that answer questions about images offer a way to automate that watch.
The team built a dataset from real classroom videos at the Banking Academy of Vietnam, with notes on the actions shown. They tested four open models, from the LLaMA and Qwen families and NVILA, on behavior questions. All four models did well enough to show promise for class analytics. The authors see a path toward tools that flag moments where a teacher may want to step in.
Catalog record
| CREATOR | Sinh Trong Vu, Hieu Trung Pham, Dung Manh Nguyen, Hieu Minh Hoang, Nhu Hoang Le, Thu Ha Pham, Tai Tan Mai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22369
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring the Application of Visual Question Answering (VQA) for Classroom Activity Monitoring Sinh Trong Vu, Hieu Trung Pham, Dung Manh Nguyen, Hieu Minh Hoang, Nhu Hoang Le, Thu Ha Pham, Tai Tan Mai Abstract Classroom behavior monitoring is a critical aspect of educational research, with significant implications for student engagement and learning outcomes. Recent advancements in Visual Question Answering (VQA) models offer promising tools for automatically analyzing complex classroom interactions from video recordings. In this paper, we investigate the applicability of several state-of-the-art open-source VQA models, including LLaMA2, LLaMA3, QWEN3, and NVILA , in the context of classroom behavior analysis. To facilitate rigorous evaluation, we introduce our BAV-Classroom-VQA dataset derived from real-world classroom video recordings at the Banking Academy of Vietnam. We present the methodology for data collection, annotation, and benchmark the performance of the selected VQA models on this dataset. Our initial experimental results demonstrate that all four models achieve promising performance levels in answering behavior-related visual questions, showcasing their potential in future classroom analytics and intervention systems. Keywords Visual Question Answering \cdot Classroom Activity Monitoring \cdot Multimodal Learning \cdot Large Language Models 1 Introduction In the current educational environment, classroom activities generate a diverse and abundant range of data, which differs significantly from traditional classroom settings that primarily rely on textual or verbal information Xiong and Wu 2020 . Among these data types, visual data plays a pivotal role in reflecting the dynamics of teaching and learning interactions. Specifically, classroom images can offer rich visual cues about students’ emotional expressions, attention levels, and their engagement with both the lesson content and their peers Xiong and Wu 2020 . However, the analysis of this type of data mainly relies on manual observation and subjective interpretation, which can lead to bias and limit the scalability of its application Blikstein 2011 . To address the challenge of improving educational quality, many recent works have sought to automate classroom observation and assessment through image-based analysis methods. Instead of relying solely on traditional indicators such as test scores or surveys, directly analyzing classroom activities through video can provide teachers with deeper insights into students’ levels of attention and engagement Bozkir et al. 2025 . Visual Question Answering (VQA) has emerged as a prominent approach to this problem, allowing systems to generate answers to questions grounded in image or video data Solawetz 2023 . Modern VQA methods have shifted from this joint encoding scheme to Vision Language Pre-training (VLP) employing transformer architectures trained on generalized tasks with large image-text pair datasets and then fine-tuned to downstream tasks like VQA Ishmam et al. 2024 , which opens up opportunities for applications in the educational domain. Integrating VQA into classroom monitoring has the potential to reduce human intervention while providing objective and real-time assessments of classroom status. This not only assists teachers in monitoring classroom activities more effectively but also generates valuable data to support the improvement of teaching methodologies. In this paper, we survey existing Visual Question Answering (VQA) models, including LLaMA2, LLaMA3, NVILA, and QWEN and assess their potential for application in classroom video analysis. Specifically, the contributions of the paper are summarized as follows: 1. Evaluate the potential of Visual Question Answering (VQA) in monitoring classroom behaviors and events, as well as in interpreting and explaining such behaviors through AI. 2. Construct a specialized VQA dataset for classroom management, namely BAV-Classroom-VQA, including behavior annotations and diverse question-answer pairs that reflect real educational scenarios. 3. Evaluate the quality and potential of state-of-the-art VQA models in the educational domain by conducting experiments on behavior recognition, event interpretation, and interactive querying in classroom videos. Figure 1: VQA Model Evaluation Framework 2 Related works 2.1 Classroom Monitoring System with computer vision In recent years, numerous studies have integrated object detection and behavior classification techniques into classroom monitoring systems. Advanced models such as YOLOv5, YOLOv7 and their refined variants - YOLO-BRA Yang et al. 2024 , YOLO-CBD Peng et al. 2025 , or YOLOv8-EMA Chen et al. 2023 have been employed in classroom and exam surveillance to identify student behavior: using mobile phone, bowing, raising hand or taking notes .These models have reported promising results, achieving [email protected] ranging from 82% to 88% on domain-specific classroom datasets, including SCB-Dataset3 Yang and Wang 2024 , highlighting their effectiveness in recognizing student behaviors in crowded and complex classroom scenarios. Despite recent advances, most existing approaches share a fundamental limitation: they process images or videos at the frame level, typically following a "detect–count–classify" pipeline. As noted in Wang and Zhu 2023 , they point out that although object detectors can localize and recognize all objects in a scene, they cannot understand the semantic relationships among these objects. This gap presents an opportunity for our research team to integrate Video Question Answering (VQA) into classroom monitoring systems, aiming not only to identify student behaviors but also to interpret and explain them through natural language queries (NLP). Such an approach paves the way for more advanced applications in analyzing educational interactions. 2.2 Visual Question Answering systems Visual Question Answering (VQA) is a key task in the field of Artificial Intelligence Uppal et al. 2022 , combining Computer Vision (CV) and Natural-Language Processing (NLP). In this task, the system is required to generate a natural language answer based on visual information from images or videos. In other words, VQA is like training the computer to not only " see " the visual elements but also to "understand" and " speak " about them when prompted with questions. Unlike traditional recognition tasks, QA requires the system to not only interpret visual content but also analyze the question and perform cross-modal reasoning to produce an appropriate answer. With the advancement of large language models, VQA has expanded from static image processing to Video Question Answering (VideoQA) - a task that requires understanding temporal dynamics and actions within video sequences. At present, numerous advanced Visual Question Answering (VQA) models have been developed, with several widely adopted models demonstrating relatively high levels of accuracy in experimental settings. Prior research has indicated that, when compared to other open-source models, LLaMA2 achieved the highest accuracy, attaining 51.7% on the EgoSchema (MC-VQA) dataset and 71.7% on the MSVU (OE-VQA) dataset Cheng et al. 2024 . LLaMA3 exhibited superior accuracy relative to other open-source models when evaluated across various datasets such as ActivityNet-QA, NextQA, and LVBench Zhang et al. 2025 . In terms of both performance and accuracy, NVILA has also received high evaluations, achieving outstanding accuracy on datasets including NeXT-QA (82.2%), MLVU (70.1%), and ActivityNet-QA (60.9%) Liu et al. 2025 . Similarly, Qwen3 has demonstrated performance surpassing that of strong models such as DeepSeek-R1 and DeepSeek-V3, and moreover, has shown competitiveness with several leading closed-source models Yang and partner 2025 . Building appropriate datasets is a key factor in the development of VQA models. Currently, popular VQA datasets such as VQA v2 and GQA mainly focus on static images Ma et al. 2024 , Sirignano and Spiliopoulos 2018 ], while datasets like TGIF-QA and MSRVTT-QA are designed for VideoQA, targeting questions about actions, events, and temporal order Jang et al. 2017a , Jang et al. 2017b .Some others, such as HowToVQA69M and ActivityNet-QA, extend the scope to instructional or action videos Yang