EchoAvatar: Real-time Generative Avatar Animation from Audio Streams 352 Conference: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers; July 19–23, 2026; Los Angeles, CA, USA Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH Conference Papers ’26), July 19–23, 2026, Los Angeles, CA, USA DOI: 10.1145/3799902.3811066 ISBN: 979-8-4007-2554-8/2026/07 CCS: Computing methodologies Animation CCS: Computing methodologies Artificial intelligence
Bohong Chen, Yumeng Li, Yinglin Xu, Youyi Zheng, Yanlin Weng, Kun Zhou · 2026
Read EchoAvatar: Real-time Generative Avatar Animation from Audio Streams 352 Conference: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers; July 19–23, 2026; Los Angeles, CA, USA Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH Conference Papers ’26), July 19–23, 2026, Los Angeles, CA, USA DOI: 10.1145/3799902.3811066 ISBN: 979-8-4007-2554-8/2026/07 CCS: Computing methodologies Animation CCS: Computing methodologies Artificial intelligence by Bohong Chen, Yumeng Li, Yinglin Xu, Youyi Zheng, Yanlin Weng, Kun Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28272.txt.
Introduction
Chen and colleagues animate avatars live from streaming speech and music, not offline clips. Their unified streaming architecture synthesizes continuous full-body motion from incremental audio, generalizes across speech and music without mode switching, and accepts intent control from language models via tool calls. It beat real-time baselines on motion quality and sync.
Catalog record
| CREATOR | Bohong Chen, Yumeng Li, Yinglin Xu, Youyi Zheng, Yanlin Weng, Kun Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72478 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28272
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EchoAvatar: Real-time Generative Avatar Animation from Audio Streams 352 Conference: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers; July 19–23, 2026; Los Angeles, CA, USA Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH Conference Papers ’26), July 19–23, 2026, Los Angeles, CA, USA DOI: 10.1145/3799902.3811066 ISBN: 979-8-4007-2554-8/2026/07 CCS: Computing methodologies Animation CCS: Computing methodologies Artificial intelligence Bohong Chen, Yumeng Li, Yinglin Xu, Youyi Zheng, Yanlin Weng, Kun Zhou Abstract . Real-time synthesis of high-fidelity 3D character motion from audio is a pivotal component for next-generation interactive avatars and virtual assistants. However, most existing approaches are limited to offline processing of complete audio sequences or are constrained to specific domains, rarely handling both speech and music effectively. In this paper, we introduce a novel framework designed to generate continuous, coherent full-body motion from streaming speech and music with low latency. Central to our approach is a unified streaming architecture capable of synthesizing continuous motion from incremental audio inputs. We employ a robust training strategy that enforces strong audio dependency, allowing the model to seamlessly generalize across conversational speech and rhythmic music without requiring explicit domain labels or mode switching. Additionally, we explored Reinforcement Learning to refine the quality of online generation. Furthermore, we bridge reactive animation with intent-driven behavior via a tool-call interface that allows upstream Large Language Models to inject explicit semantic control. By combining this controllability with stream audio-driven synthesis, our framework serves as a plug-and-play solution for transforming voice agents into interactive humanoid avatars. Extensive experiments demonstrate that our method outperforms state-of-the-art real-time baselines in motion quality and synchronization while maintaining the flexibility required for live deployment. Our code, pre-trained models, and videos are available at https://robinwitch.github.io/EchoAvatar-Page. Figure 1. Given streaming audio input, our method generates avatar animation in a streaming manner. The four poses shown above are sampled from a continuous motion sequence driven by the audio stream. † † cc-license: by 1. Introduction The rapid evolution of Large Language Models (LLMs) and Voice Agents has enabled fluid, natural dialogue. However, while audio fidelity is near-human, visual embodiment lacks the responsiveness required for genuine interaction, necessitating high-fidelity 3D motion generation directly from streaming audio with low latency. Existing approaches to audio-driven motion synthesis fall short of meeting this challenge for two primary reasons. First, most state-of-the-art methods ( 51 ; 105 ; 12 ) are designed for offline processing, requiring complete audio sequences as input before generating motion. This architectural constraint introduces unacceptable latency for real-time interactive applications. Second, existing methods are typically domain-specific, handling either speech or music but rarely both. This fragmentation necessitates complex model switching, limiting their applicability to general-purpose voice agents that must handle diverse acoustic inputs uniformly. In this paper, we present a unified framework for real-time, streaming avatar animation that addresses these limitations. This system takes live audio stream and generates motion in real-time, featuring a causal motion tokenizer for high-quality auto-regressive synthesis and a specially designed training strategy for unified learning. Regarding the tokenizer, while recent works have explored causal architectures via causal convolutions ( 37 ; 87 ) , we find that pure convolutional approaches often lack expressiveness and suffer from reconstruction artifacts. We instead propose an attention-based causal motion tokenizer with auxiliary kinematic losses, achieving superior generation quality and streaming capability. Regarding the training strategy, we identify that optimization dynamics within a unified motion space often weaken audio conditioning, leading to catastrophic failure in task alignment. We address this via a hierarchical token corruption strategy that enhances audio conditioning, enabling the model to uniformly learn conversational gestures and rhythmic dance without explicit domain labels. Furthermore, experiments reveal a synergistic effect where the integration of diverse motion domains mutually reinforces generation fidelity. Beyond real-time generation, our system is designed for practical deployment within modern voice agent ecosystems. It operates as a plug-and-play module, accepting audio streams from diverse sources ranging from web browsers to AI conversational platforms. Furthermore, we introduce a tool-call interface that enables upstream systems, such as Large Language Models, to interleave explicit semantic actions with implicit audio-driven motion, bridging the gap between purely reactive audio-driven animation and controllable, intent-driven behavior. To further align real-time generation with human preferences, we explore Reinforcement Learning (RL) by investigating both reward-model-based strategies using Group Relative Policy Optimization (GRPO) and human-annotation-based strategies using Direct Preference Optimization (DPO). We demonstrate measurable improvements in perceived quality and provide an analysis of applying RL to online auto-regressive motion generation for future research. Our primary contributions are: • A unified streaming architecture that leverages attention-based causal tokenization to synthesize continuous, high-fidelity motion from streaming speech and music with low latency. • A robust training curriculum utilizing Hierarchical Token Corruption to enable synergistic learning across diverse domains, boosting performance on individual tasks, alongside an exploration of RL strategies (GRPO/DPO) to enhance perceived generation quality. • A deployable plug-and-play system that integrates with voice agents, supporting both implicit audio-driven animation and explicit semantic control via tool calls. 2. Related Work 2.1. Co-speech Gesture Generation The trajectory of co-speech gesture generation reflects a fundamental paradigm shift from explicit, rule-based heuristics to implicit, data-driven synthesis. Early frameworks ( 39 ; 10 ; 42 ; 43 ; 9 ) relied on rigid production rules and manual linguistic mappings, which offered controllability but lacked kinematic naturalness. The advent of deep learning initially spurred deterministic regression approaches ( 40 ; 52 ; 98 ; 110 ; 30 ) ; however, by modeling the modal average of plausible motions, these methods frequently suffered from “mean-pose convergence”, resulting in over-smoothed and under-articulated output. To address the inherent stochasticity and one-to-many ambiguity of the speech-to-gesture mapping, the field has pivoted toward probabilistic generative modeling. This landscape encompasses Normalizing Flows ( 2 ; 96 ) for explicit density estimation, Variational Autoencoders (VAEs) ( 25 ; 44 ; 77 ) for continuous latent structuring, and Vector Quantized (VQ) frameworks ( 95 ; 4 ; 56 ; 55 ; 97 ; 59 ) that learn discrete motion codebooks. Diffusion Probabilistic Models based approaches ( 3 ; 5 ; 93 ; 19 ; 103 ; 62 ; 91 ) excelling at modeling complex distributions via iterative denoising and MLLM ( 14 ; 17 ; 35 ; 49 ) unifying 3D human motion with text and speech in a shared latent space. Among these, ConvoFusion ( 63 ) extends diffusion-based generation by enabling gesture emphasis on specific words. Teller ( 108 ) proposes a real-time audio-driven portrait talking head system. ACRNN ( 112 ) is the first method capable of generating arbitrarily long motions in real time with stability. While many approaches ( 15 ; 53 ) claim real-time capability for audio-driven body motion generation, they merely achieve generation speeds faster than playback speed under the assumption of full audio context availability. True streaming scenarios—where audio is incrementally received and motion is progressively generated—remain largely unexplored. 2.2. Multimodal Motion Synthesis Motion synthesis research has significantly expanded its scope by integrating diverse control signals beyond audio. These range from semantic text descriptions ( 102 ; 82 ; 60 ; 23 ; 6 ) and spatial trajectory constraints ( 88 ; 84 ; 109 ) to physical interaction states ( 54 ; 58