U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation
Xiang Deng, Feng Gao, Yong Zhang, Youxin Pang, Xu Xiaoming, Zhuoliang Kang, Xiaoming Wei, Yebin Liu · 2026
Read U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation by Xiang Deng, Feng Gao, Yong Zhang, Youxin Pang, Xu Xiaoming, Zhuoliang Kang, Xiaoming Wei, Yebin Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-23739.txt.
Description
Xiang Deng and seven co-authors present U-Mind, a unified method for real-time multi-modal chat. Multi-modal chat is central to bodied agents. Current systems are limited to single-mode making or suffer weakened thinking and poor cross-modal matching.
U-Mind is the first system for multi-modal chat, building language, speech, motion, and video making within a single live loop. At its core, a Unified Matching and Thinking Method tackles two hurdles: lifting cross-modal syncing via segment-wise matching, and keeping thinking through practice-driven learning. During use, a text-first reading line performs internal chain-of-thought planning then time-synced making. A video drawing method tied to pose and speech gives lively, synced feedback. Tests show state-of-the-art scores on multi-modal chat tasks: question answers, order obeying, and motion making.
Catalog record
| CREATOR | Xiang Deng, Feng Gao, Yong Zhang, Youxin Pang, Xu Xiaoming, Zhuoliang Kang, Xiaoming Wei, Yebin Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-23739