Single-agent vs. Multi-agents for Automated Video Analysis of On-Screen Collaborative Learning Behaviors
Likai Peng, Shihui Feng · 2026
Read Single-agent vs. Multi-agents for Automated Video Analysis of On-Screen Collaborative Learning Behaviors by Likai Peng, Shihui Feng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03631.txt.
Introduction
Peng and Feng compare single-agent and multi-agent vision language models for auto coding of screen clips in group learning, using the ICAP framework. They test leading closed-source models and an open-source model in single- and multi-agent settings. These include a three-agent workflow system and a self-deciding system inspired by ReAct. Both multi-agent frameworks beat single models on scene and action spotting, with the workflow agent best on scenes and the self-deciding agent best on actions.
Catalog record
| CREATOR | Likai Peng, Shihui Feng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30973 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03631
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Single-agent vs. Multi-agents for Automated Video Analysis of On-Screen Collaborative Learning Behaviors Likai Peng, Shihui Feng Abstract On-screen learning behavior provides valuable insights into how students seek, use, and create information during learning. Analyzing on-screen behavioral engagement is essential for capturing students’ cognitive and collaborative processes. The recent development of Vision Language Models (VLMs) offers new opportunities to automate the labor-intensive manual coding often required for multimodal video data analysis. In this study, we compared the performance of both leading closed-source VLMs (Claude-3.7-Sonnet, GPT-4.1) and open-source VLM (Qwen2.5-VL-72B) in single- and multi-agent settings for automated coding of screen recordings in collaborative learning contexts based on the ICAP framework. In particular, we proposed and compared two multi-agent frameworks: 1) a three-agent workflow multi-agent system (MAS) that segments screen videos by scene and detects on-screen behaviors using cursor-informed VLM prompting with evidence-based verification; 2) an autonomous-decision MAS inspired by ReAct that iteratively interleaves reasoning, tool-like operations (segmentation/ classification/ validation), and observation-driven self-correction to produce interpretable on-screen behavior labels. Experimental results demonstrated that the two proposed MAS frameworks achieved viable performance, outperforming the single VLMs in scene and action detection tasks. It is worth noting that the workflow-based agent achieved best on scene detection, and the autonomous-decision MAS achieved best on action detection. This study demonstrates the effectiveness of VLM-based Multi-agent System for video analysis and contributes a scalable framework for multimodal data analytics. 1 Introduction Understanding students’ behavioral engagement is critical for understanding how learning occurs and for developing effective teaching and intervention strategies [ 15 ] . To capture, analyze and visualize this engagement, researchers increasingly rely on multimodal learning process data, which integrates diverse sources such as interaction logs, oral recordings, written outputs, eye tracking data, spatial movements, and facial expressions [ 22 , 12 ] . Although multimodal data has been widely used to analyze engagement, screen recordings, capturing both on-screen actions and contextual scenes, remain underexplored [ 23 ] . Screen recordings offer distinct advantages over traditional data sources [ 25 ] . Unlike log data, which captures discrete events (e.g., clicks or page visits), screen recording data show more detailed and fine-grained records of on-screen actions and learning context. However, current analyses of screen recordings heavily rely on manual coding, which becomes impractical for large datasets or complex coding schemes (e.g., multidimensional collaboration behaviors) [ 10 ] . Recent advances in computer vision, exemplified by the Computer Vision for Position Estimation (CVPE) framework [ 18 ] , have demonstrated the feasibility of scalable, privacy-preserving extraction of socio-spatial behaviors from video data. However, automated approaches to on-screen action recognition in educational contexts are still in their early stages of development. Vision language model (VLM) has demonstrated a high level of zero-shot performance in a variety of visual tasks including action recognition and understanding [ 4 ] . While VLM offers new possibilities for understanding onscreen behavior, it still lags behind in fine-grained behavior recognition and temporal understanding [ 28 ] . Multi-agent collaboration can improve video understanding by enabling complementary specialists (e.g., grounding and vision agents) to gather evidence and support multi-step reasoning, outperforming non-agent baselines [ 19 ] . This study aims to assess different agent systems using VLMs in on-screen behavior analysis, including workflow-based and autonomous decision-making agent systems. We develop a new method leveraging VLMs and multi-agent techniques to automatically analyze learning behavior using on-screen data based on the ICAP framework [ 9 ] . Existing AI in education research has predominantly centered on LLMs, by integrating LLMs into the learning process (e.g., AI-supported peer feedback) or assessment [ 13 ] . However, the use of VLMs holds considerable potential to enhance understanding of multimodal data in educational contexts, as they can facilitate deeper understanding of learning processes by integrating visual and textual information for tasks such as image captioning, visual question answering, and visual search [ 34 ] . This study explored the following research questions. RQ1: To what extent can single VLMs (commercial: GPT-4.1, Claude-3.7-Sonnet; open-source: Qwen2.5-VL-72B) accurately code onscreen behaviors from video in a few-shot setting? RQ2: What was the performance of multi-agent orchestration strategies (workflow-based vs. autonomous decision-making) on coding onscreen behaviors compared to single-agent baselines? 2 Related Work 2.1 On-Screen Learning Behavior The ICAP framework categorizes learner engagement into Passive, Active, Constructive and Interactive modes offering a theoretical lens for interpreting observable behaviors as cognitive engagement [ 9 ] . Applied across diverse contexts from video-based STEM activity recognition to robot-supported collaborative learning—ICAP has proven effective for analyzing on-screen behaviors and inferring engagement level [ 17 , 5 ] . Prior work has examined on-screen behaviors at both individual and group levels. Sundar compared six different on-screen actions (click-to-download, drag, mouseover, slide, zoom, and 3D carousel) and found that certain information-based actions, such as mouseover, provided a more positive user experience compared to others like zoom, while some navigation actions, such as slide, appeared to be more effective than alternatives like drag [ 25 ] . In online learning environments, Cheung used verbal exchanges and on-screen interactions among three pairs of learners to explore collaborative writing patterns and how they leveraged semiotic resources and multimodal elements, identifying that “Text type and revision” and “Internet search” were the most frequent on-screen activities in the collaborative writing process [ 8 ] . Oh and Sundar further linked interaction frequency (clicks, drags) to learners’ perceptions of online content [ 20 ] . Despite these advances, existing studies mainly analyzed on-screen behavior within a constrained learning platform. Few prior studies have investigated students’ on-screen actions across platforms within a learning task, such as generative AI applications, web search engines, and shared group documents. This is critical for understanding how students navigate and integrate diverse digital resources in authentic, multi-tool learning environments. 2.2 Generative AI-assisted Video Analysis VLMs for Video Analysis VLMs integrate computer vision and natural language processing to interpret multimodal content, demonstrating strong potential and capabilities in video analytics, including multimodal emotion recognition and collaborative activities [ 26 ] . Gao et al. proposed T3-Agent for tuning multi-modal agents, which automatically synthesizes tool-usage data and finetunes VLMs as controllers for sophisticated multi-modal reasoning which improved VLM reasoning capabilities by 20% through automated tool-usage data [ 14 ] . In educational applications, VLMs have been used to extract and interpret images and videos. Alahmadi and Alshangiti demonstrated that LLMs such as GPT-4V and Gemini outperform traditional optical character recognition (OCR) engines in transcribing source code from screencasts across various video qualities. These results indicate the robustness and advanced capabilities of VLMs in diverse educational scenarios [ 1 ] . However, significant performance gaps remain—AutoEval-Video benchmarks reveal VLMs achieve only 32.2% accuracy compared to 72.8% human performance in video question answering, particularly struggling with temporal and dynamic comprehension critical for educational contexts [ 7 ] . Multi-agent System (MAS) for Video Analysis Multi-agent systems mitigate VLM’s limitations through distributed coordination [ 11 ] . Educational applications leverage agent specialization: Jiang et al.’s von Neumann framework decomposes curriculum design into modular tasks (knowledge graphs, resource matching, difficulty calibration) processed by specialized agents [ 16 ] . This close collaboration enables the creation of adaptive, dynamic course plans that are tailored to individual learning needs. Yang et