Learning Modal-mixed Chain-of-thought Reasoning with Latent Embeddings
Yifei Shao, Kun Zhou, Ziming Xu, Mohammad Atif Quamar · 2026
Read Learning Modal-mixed Chain-of-thought Reasoning with Latent Embeddings by Yifei Shao, Kun Zhou, Ziming Xu, Mohammad Atif Quamar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00574.txt.
Description
Yifei Shao and colleagues extend chain-of-thought reasoning beyond language to vision problems, where text-only steps fail since key states are visual. Their modal-mixed chain of thought interleaves text tokens with visual sketches as latent embeddings, using the vision-language model as encoder. A diffusion-based latent decoder, fired by a control token, carries fine detail while the model sets high-level intent. Training mixes supervised fine-tuning on interleaved traces with reinforcement learning on switching modalities, beating language-only and CoT methods on 11 tasks.
Catalog record
| CREATOR | Yifei Shao, Kun Zhou, Ziming Xu, Mohammad Atif Quamar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45602 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.00574