VTutor: An Animated Pedagogical Agent SDK that Provide Real Time Multi-Model Feedback
Eason Chen, Chenyu Lin, Yu-Kai Huang Xinyi Tang Aprille Xi Jionghao Lin Kenneth Koedinger · 2025
Read VTutor: An Animated Pedagogical Agent SDK that Provide Real Time Multi-Model Feedback by Eason Chen, Chenyu Lin, Yu-Kai Huang Xinyi Tang Aprille Xi Jionghao Lin Kenneth Koedinger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06676.txt.
Introduction
Eason Chen and co-authors published a research paper in 2025 on an open-source kit for building animated virtual tutors that speak and react in real time. Talking-head tutors promise personal feedback at scale, but older systems sound scripted, move stiffly, or cost too much to build. Their kit plugs a language model's words into speech and a lip-synced cartoon character.
An anime-like look sidesteps the creepy near-human valley. In a fifty-person study, the tutor beat a leading talking-head baseline on sync accuracy, naturalness, expressiveness, and overall preference. The authors invite the community to add characters and classroom showcases, aiming to spread cheap, likable AI tutors across web courses.
Catalog record
| CREATOR | Eason Chen, Chenyu Lin, Yu-Kai Huang Xinyi Tang Aprille Xi Jionghao Lin Kenneth Koedinger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VTutor: An Animated Pedagogical Agent SDK that Provide Real Time Multi-Model Feedback Eason Chen, Chenyu Lin, Yu-Kai Huang Xinyi Tang Aprille Xi Jionghao Lin Kenneth Koedinger Abstract Pedagogical Agents (PAs) show significant potential for boosting student engagement and learning outcomes by providing adaptive, on-demand support in educational contexts. However, existing PA solutions are often hampered by pre-scripted dialogue, unnatural animations, uncanny visual realism, and high development costs. To address these gaps, we introduce VTutor, an open-source SDK leveraging lightweight WebGL, Unity, and JavaScript frameworks. VTutor receives text outputs from a large language model (LLM), converts them into audio via text-to-speech, and then renders a real-time, lip-synced pedagogical agent (PA) for immediate, large-scale deployment on web-based learning platforms. By providing on-demand, personalized feedback, VTutor strengthens students’ motivation and deepens their engagement with instructional material. Using an anime-like aesthetic, VTutor alleviates the uncanny valley effect, allowing learners to engage with expressive yet comfortably stylized characters. Our evaluation with 50 participants revealed that VTutor significantly outperforms the existing talking-head approaches (e.g., SadTalker) on perceived synchronization accuracy, naturalness, emotional expressiveness, and overall preference. As an open-source project, VTutor welcomes community-driven contributions—from novel character designs to specialized showcases of pedagogical agent applications—that fuel ongoing innovation in AI-enhanced education. By providing an accessible, customizable, and learner-centered PA solution, VTutor aims to elevate human-AI interaction experience in education fields, ultimately broadening the impact of AI in learning contexts. The demo link to VTutor is at https://vtutor-aied25.vercel.app . 1 Introduction and Background AI in Education (AIED) aims to enhance learning processes and outcomes by leveraging artificial intelligence techniques [ 21 ] such as intelligent tutoring systems [ 23 ] , adaptive assessments [ 19 ] , programming [ 3 ] , and chatbot [ 2 ] . While large language models (LLMs) now excel at generating personalized content and responding to learners’ open-ended queries [ 4 , 5 , 25 , 1 , 15 ] , many of these systems are constrained by purely text-based or voice-based interfaces. Pedagogical Agents (PAs) offers an opportunity for a more human-like interaction experience with LLMs [ 22 , 16 , 14 ] , delivering immediate, adaptive guidance via facial expressions, gestures, and voice [ 10 , 24 , 17 , 9 ] . However, existing PA frameworks often rely on scripted interactions [ 20 , 8 ] , utilize static or non-synchronized facial expressions [ 14 ] , high development cost, and long latency [ 16 , 26 ] . These constraints lead to interaction delays, unnatural facial movements, and overall lower engagement, thereby limiting the development of dynamic, learner-centered AI tutoring experiences. As a result, these PAs cannot reliably provide responsive feedback, emotional support, and adaptive scaffolding—key features that enhance the tutoring process. Moreover, when a PA design becomes overly humanlike, it risks triggering the “uncanny valley” phenomenon [ 7 , 22 ] , wherein users feel discomfort toward entities that appear almost, but not exactly, human. This effect can discourage learners from interacting with the PA, undercutting its potential to facilitate meaningful learning outcomes. Consequently, there is a growing need for an integrated solution that aligns robust LLM-driven dialogue capabilities with visually appealing, synchronized, and scalable PA tools, all while avoiding the uncanny valley effect to preserve learner engagement and trust. Meanwhile, virtual avatars have increasingly become a presence in real-world applications, particularly in the entertainment field, where many content creators opt to engage with their audiences using digital personas instead of revealing their real faces. These creators, known as VTubers [ 12 , 6 ] , leverage face tracking and motion capture technologies to animate their avatars in real-time by 3D engine like Unity, enabling expressive and interactive performances. The aesthetic of VTuber avatars is heavily influenced by anime culture, incorporating cuteness and charm elements that make them visually appealing and emotionally engaging for viewers [ 12 , 6 ] . These stylistic choices not only enhance relatability but also circumvent the uncanny valley effect, maintaining user comfort and engagement. Our Contribution: This paper introduces VTutor , an open-source Software Development Kit (SDK) designed to streamline the development and deployment of animated pedagogical agents (APAs) in educational settings. VTutor enhances human-AI interaction by enabling real-time, engaging, and emotionally expressive AI-driven tutors. Specifically, VTutor contributes: 1. A Generative AI-Driven Interactive Agent: By integrating output from LLMs with animated pedagogical agents, VTutor facilitates real-time, personalized, and adaptive interaction experiences. 2. Customizable Characters: VTutor adopts anime-style characters that strike a balance between realism and abstraction, avoiding the uncanny valley while enhancing engagement across diverse learners. 3. A Seamless, Scalable, and Developer-Friendly SDK: The lightweight browser-compatible architecture ensures that VTutor can be effortlessly embedded into existing educational platforms with minimal setup. By addressing key challenges in traditional APAs—such as lack of real-time adaptability, poor lip synchronization, and high computational overhead—VTutor paves the way for more immersive and accessible AI-driven educational tools. 2 VTutor System Implementation Figure 1: Screenshot of the VTutor SDK demo at https://vtutor-aied25.vercel.app/sdk , the few lines of codes in the image are what developers need to do to add VTutor to their own website. VTutor is the animated avatar presented at the bottom right. VTutor is composed of the following components: (1) Large Language Model Integration. VTutor presents AI-generated text responses by integrating various LLMs (e.g., GPT, Claude, Deepseek). Educators can customize prompts or knowledge bases to align PA interactions with learning objectives, enabling flexible, contextually adaptive dialogues. (2) Text-to-Speech (TTS). VTutor accepts audio in .wav format from major TTS services (e.g., Azure, ElevenLabs). This enables high-quality, natural speech that resonates with learners. During run-time, the system retrieves newly generated text from the LLM, converts it to audio via a TTS API, and sends it to the Unity engine for real-time facial expression and mouth rendering. (3) LipSync Module. A Unity lip-sync library [ 13 ] analyzes incoming audio to detect phoneme sequences. Corresponding mouth-blend shapes are applied to a 3D or 2D avatar, achieving near real-time alignment with an average latency of under one second for short utterances. This approach yields more accurate and smooth mouth movements compared to naive keyframe or jaw-only animations. (4) WebGL-Based Rendering. VTutor packages the Unity environment into a WebGL build, allowing easy embedding in any web page by iframe. Communication between the web page and the embedded PA occurs via an open-sourced software development kit (SDK) library, enabling dynamic control and fine-grained interactivity (e.g., changing gestures, expressions, or switching avatars). 3 Evaluation Method and Results We invited participants to compare VTutor with talking-head pedagogical agents generated using SadTalker [ 26 ] , following the methodology described by [ 16 ] . To minimize potential appearance biases, we used the same VTutor agent design to create the SadTalker talking head. The full evaluation materials can be found at https://vtutor-aied25.vercel.app/evaluation . Our experiment received approval from the Institutional Review Board (IRB). We recruited 50 participants for evaluation through convenience sampling via Prolific. The participants were between 20 and 50 years old, consisting of 30 males and 20 females. We conducted the experiment online, where participants watched the videos on YouTube and responded to the questions using Google Forms. The videos featured VTutor and Sad Talker speaking, but throughout the entire experiment, they were referred