U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation

Xiang Deng, Feng Gao, Yong Zhang, Youxin Pang, Xu Xiaoming, Zhuoliang Kang, Xiaoming Wei, Yebin Liu · 2026

Read U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation by Xiang Deng, Feng Gao, Yong Zhang, Youxin Pang, Xu Xiaoming, Zhuoliang Kang, Xiaoming Wei, Yebin Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-23739.txt.

Description

Xiang Deng and seven co-authors present U-Mind, a unified method for real-time multi-modal chat. Multi-modal chat is central to bodied agents. Current systems are limited to single-mode making or suffer weakened thinking and poor cross-modal matching.

U-Mind is the first system for multi-modal chat, building language, speech, motion, and video making within a single live loop. At its core, a Unified Matching and Thinking Method tackles two hurdles: lifting cross-modal syncing via segment-wise matching, and keeping thinking through practice-driven learning. During use, a text-first reading line performs internal chain-of-thought planning then time-synced making. A video drawing method tied to pose and speech gives lively, synced feedback. Tests show state-of-the-art scores on multi-modal chat tasks: question answers, order obeying, and motion making.

Catalog record

CREATORXiang Deng, Feng Gao, Yong Zhang, Youxin Pang, Xu Xiaoming, Zhuoliang Kang, Xiaoming Wei, Yebin Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-23739

Browse the full library · About the Tower of Babel