Thinking in Text and Images: Interleaved Vision–Language Reasoning Traces for Long-Horizon Robot Manipulation
Jinkun Liu, Haohan Chi, Lingfeng Zhang, Yifan Xie, YuAn Wang, Long Chen, Hangjun Ye, Xiaoshuai Hao, Wenbo Ding · 2026
Read Thinking in Text and Images: Interleaved Vision–Language Reasoning Traces for Long-Horizon Robot Manipulation by Jinkun Liu, Haohan Chi, Lingfeng Zhang, Yifan Xie, YuAn Wang, Long Chen, Hangjun Ye, Xiaoshuai Hao, Wenbo Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00438.txt.
Description
Jinkun Liu and 8 co-authors teach robot policies to think in text and images at once. Long manipulation tasks need plans that are logically sound and geometrically grounded. Text-only reasoning orders the steps but misses spatial limits, while image prediction shows geometry but stays local and vague. Latent plans, hidden inside the network, cannot be inspected at all.
Their framework builds an explicit trace that alternates written subgoals with visual keyframes across the whole task. One multimodal model generates the trace from the opening view and the instruction, caches it, and a closed-loop controller then acts under its guidance. Training traces are manufactured by slicing demonstrations into stages and captioning each stage automatically. On long-horizon simulation benchmarks the interleaved trace far outperformed text-only, image-only, and no-trace variants, though the authors note stale or wrong global plans remain a weakness.
Catalog record
| CREATOR | Jinkun Liu, Haohan Chi, Lingfeng Zhang, Yifan Xie, YuAn Wang, Long Chen, Hangjun Ye, Xiaoshuai Hao, Wenbo Ding |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00438