From Natural Alignment to Conditional Controllability in Multimodal Dialogue
Zeyu Jin, Songtao Zhou, Haoyu Wang, Minghao Tian, Kaifeng Yun, Zhuo Chen, Xiaoyu Qin, Jia Jia · 2026
Read From Natural Alignment to Conditional Controllability in Multimodal Dialogue by Zeyu Jin, Songtao Zhou, Haoyu Wang, Minghao Tian, Kaifeng Yun, Zhuo Chen, Xiaoyu Qin, Jia Jia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29162.txt.
Description
Jin and co-authors work on controllable multimodal dialogue generation. Current systems make realistic speech or vision alone. Controlling both together stays hard. The team builds an annotation pipeline that mines dialogues from movies and TV series. Their MM-Dia dataset holds over 360 hours and 54,700 dialogues. Fine-grained labels capture interaction style across speech, vision, and text. A smaller benchmark, MM-Dia-Bench, tests cross-modal style consistency. Training on MM-Dia improved fine-grained control in their experiments.
Catalog record
| CREATOR | Zeyu Jin, Songtao Zhou, Haoyu Wang, Minghao Tian, Kaifeng Yun, Zhuo Chen, Xiaoyu Qin, Jia Jia |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39111 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.29162